getTracks... still not quite there

- JustSomeOldJoe disse...
- Assinante
- Jun 5 2009, 20h27
getTracks... still not quite there

The current behaviour of library.getTracks is to return only those tracks that have been played at least once. This is obviously problematic in terms of being able to fetch a user's entire lib with a single API call.

If I'm in discovery mode, I might be tagging and/or adding tracks to a playlist while listening, yet "not" listening to the entire track. Technically, those tracks become part of my lib, yet they're not retrieved by getTracks since their play count remains zero.

As a result, I'm forced to combine getTracks with both tag and playlist fetches, combine all fetches, remove dups, and create a union to finally arrive at a user's complete library. Seems to be far more work than should be required (not to mention, a bigger service hit) to simply retrieve all tracks in a user's lib.

The ability to include those tracks that have been added to a user's lib yet not played within the scope of a getTracks fetch would be very helpful. Adding a simple flag parm ("fetch all vs. fetch only played") and defaulting that parm would accomplish the task without breaking existing API code.

[quote][b][user]JustSomeOldJoe[/user][/b] disse: The current behaviour of library.getTracks is to return only those tracks that have been played at least once. This is obviously problematic in terms of being able to fetch a user's entire lib with a single API call. If I'm in discovery mode, I might be tagging and/or adding tracks to a playlist while listening, yet "not" listening to the entire track. Technically, those tracks become part of my lib, yet they're not retrieved by getTracks since their play count remains zero. As a result, I'm forced to combine getTracks with both tag and playlist fetches, combine all fetches, remove dups, and create a union to finally arrive at a user's complete library. Seems to be far more work than should be required (not to mention, a bigger service hit) to simply retrieve all tracks in a user's lib. The ability to include those tracks that have been added to a user's lib yet not played within the scope of a getTracks fetch would be very helpful. Adding a simple flag parm ("fetch all vs. fetch only played") and defaulting that parm would accomplish the task without breaking existing API code.[quote]
Editado por JustSomeOldJoe em Nov 6 2009, 4h02

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Jun 9 2009, 21h08
No comments on this one at all?

* Taken from LFM's own API Doc:

library.getTracks

A paginated list of all the tracks in a user's library, with play counts and tag counts.

The API description is incorrect. Tracks with zero playcount "are" technically part of my library if I've: a. tagged 'em; b. added them to "my lib"; or c. added them to a playlist, regardless of whether they've been played or not. They "are" listed under the "Library" section of my account. Yet, library.getTracks DOES NOT return these tracks... it returns only tracks with a playcount greater than zero.

I have to believe this is a bug. Possibly an equi-join in the API vs. an outer-join used for the internal lib fetch. If not, then it's an API doc error.

A clarification would be much appreciated!

[quote][b][user]JustSomeOldJoe[/user][/b] disse: No comments on this one at all? * Taken from LFM's own API Doc: library.getTracks A paginated list of [b]all the tracks in a user's library[/b], with play counts and tag counts. The API description is incorrect. Tracks with zero playcount "are" technically part of my library if I've: a. tagged 'em; b. added them to "my lib"; or c. added them to a playlist, regardless of whether they've been played or not. They "are" listed under the "Library" section of my account. Yet, library.getTracks DOES NOT return these tracks... it returns only tracks with a playcount greater than zero. I have to believe this is a bug. Possibly an equi-join in the API vs. an outer-join used for the internal lib fetch. If not, then it's an API doc error. A clarification would be much appreciated![quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Jul 1 2009, 8h19
I'm going to bump this one final time in hopes of getting a response. Just to clarify, the ramifications are:

* Any external service that states it can import LFM track data and does so via a call to library.getTracks() will NOT work as intended and may yield inconsistent results.

* Any 3rd party tool that attempts to integrate LFM data by pulling the LFM dataset via getTracks(), then, subsequently, merging the results with its own dataset will NOT work as intended and may yield inconsistent results.

* All data that has been explicitly added to a user's library via "Love Track", "Add Track", "Tag Track", and/or "Add Track to Playlist" will NOT be pulled via the library.getTracks() call "unless" the track has actually been played (scrobbled).

[quote][b][user]JustSomeOldJoe[/user][/b] disse: I'm going to bump this one final time in hopes of getting a response. Just to clarify, the ramifications are: [b]* Any external service that states it can import LFM track data and does so via a call to library.getTracks() will NOT work as intended and may yield inconsistent results. * Any 3rd party tool that attempts to integrate LFM data by pulling the LFM dataset via getTracks(), then, subsequently, merging the results with its own dataset will NOT work as intended and may yield inconsistent results. * All data that has been explicitly added to a user's library via "Love Track", "Add Track", "Tag Track", and/or "Add Track to Playlist" will NOT be pulled via the library.getTracks() call "unless" the track has actually been played (scrobbled). [/b][quote]

Permalink Citar
- discospit disse...
- Usuário
- Out 4 2009, 17h43
this does seem rather silly

and i for one agree that getTracks should return ALL tracks in a user's library. also id love it if there was a tag parameter to the call

[quote][b][user]discospit[/user][/b] disse: and i for one agree that getTracks should return ALL tracks in a user's library. also id love it if there was a tag parameter to the call[quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Out 4 2009, 19h10
Surprised this one popped back up.

Yup, using "Library" as a metaphor should make the answer obvious. If a librarian purchases a book, creates a card, tags the book and adds it to the shelves, it's part of the library.

Using the current implementation, the book wouldn't be considered part of the library until someone actually read it... or at leat 50% of it.

[quote][b][user]JustSomeOldJoe[/user][/b] disse: Surprised this one popped back up. Yup, using "Library" as a metaphor should make the answer obvious. If a librarian purchases a book, creates a card, tags the book and adds it to the shelves, it's part of the library. Using the current implementation, the book wouldn't be considered part of the library until someone actually read it... or at leat 50% of it.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Out 4 2009, 20h43
You're right, this does seem like a bug. I'll make sure this gets investigated.

[quote][b][user]jwheare[/user][/b] disse: You're right, this does seem like a bug. I'll make sure this gets investigated.[quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Out 4 2009, 21h05
jwheare said:
You're right, this does seem like a bug. I'll make sure this gets investigated.

jw... that would be mucho appreciated!

However, isn't it Sunday "night" in your neck of the woods? Shouldn't you be... I dunno... SLEEPING?

[quote][b][user]JustSomeOldJoe[/user][/b] disse: [quote][b][user]jwheare[/user][/b] said: You're right, this does seem like a bug. I'll make sure this gets investigated.[/quote] jw... that would be mucho appreciated! However, isn't it Sunday "night" in your neck of the woods? Shouldn't you be... I dunno... SLEEPING?[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Out 4 2009, 23h23
JustSomeOldJoe said:
However, isn't it Sunday "night" in your neck of the woods? Shouldn't you be... I dunno... SLEEPING?

Didn't say I'd investigate it right away :) However it is now filed as an issue in our internal bug tracker.

[quote][b][user]jwheare[/user][/b] disse: [quote][b][user]JustSomeOldJoe[/user][/b] said: However, isn't it Sunday "night" in your neck of the woods? Shouldn't you be... I dunno... SLEEPING?[/quote] Didn't say I'd investigate it right away :) However it is now filed as an issue in our internal bug tracker.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 2 2009, 16h16
Hey JustSomeOldJoe, could you confirm that this is still happening. A change was made to this service around the end of July that called a different method to get the tracks. I have a hunch that this also fixed the issue you're describing here.

If you're still seeing the problem, could you give me a URL I can check to confirm. Thanks.

[quote][b][user]jwheare[/user][/b] disse: Hey JustSomeOldJoe, could you confirm that this is still happening. A change was made to this service around the end of July that called a different method to get the tracks. I have a hunch that this also fixed the issue you're describing here. If you're still seeing the problem, could you give me a URL I can check to confirm. Thanks.[quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Nov 2 2009, 19h13
Nope... still broken.

You can run the call from the samp API page up against your own account and see no 0 playcount tracks returned. The call returns tracks in DESC PLAYCOUNT order, so just jump to the last page and look at the last rec's playcount (it'll be 1).

Here's the link for your own lib:

http://ws.audioscrobbler.com/2.0/?method=library.gettracks&api_key=b25b959554ed76058ac220b7b2e0a026&user=jwheare&page=242

I wrote a kluge a while back to bypass the randomizer and force play of the zero playcounts, so the overall number in my own lib is down quite dramatically. There's still a few... just closer to 1 to 2% now rather then 10-20%.

If you want samps from my own lib, just click on Library, jump to the last page, and grab any artist listed with "0 Plays". If you run the call against my own lib, you won't see the 0 plays included.

[quote][b][user]JustSomeOldJoe[/user][/b] disse: Nope... still broken. You can run the call from the samp API page up against your own account and see no 0 playcount tracks returned. The call returns tracks in DESC PLAYCOUNT order, so just jump to the last page and look at the last rec's playcount (it'll be 1). Here's the link for your own lib: http://ws.audioscrobbler.com/2.0/?method=library.gettracks&api_key=b25b959554ed76058ac220b7b2e0a026&user=jwheare&page=242 I wrote a kluge a while back to bypass the randomizer and force play of the zero playcounts, so the overall number in my own lib is down quite dramatically. There's still a few... just closer to 1 to 2% now rather then 10-20%. If you want samps from my own lib, just click on Library, jump to the last page, and grab any artist listed with "0 Plays". If you run the call against my own lib, you won't see the 0 plays included.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 3 2009, 10h17
Thanks, I'll take a closer look at this today.

[quote][b][user]jwheare[/user][/b] disse: Thanks, I'll take a closer look at this today.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 3 2009, 14h08
OK I believe this is now fixed. The current behaviour actually does show you tracks that you've manually added, but it assigns them a playcount of 1, but tracks that are only tagged are indeed being left out.

This fix should be going out along with our site release tomorrow.

Sorry this took so long to spot and fix, it was a pretty hairy bug.

[quote][b][user]jwheare[/user][/b] disse: OK I believe this is now fixed. The current behaviour actually does show you tracks that you've manually added, but it assigns them a playcount of 1, but tracks that are only tagged are indeed being left out. This fix should be going out along with our site release tomorrow. Sorry this took so long to spot and fix, it was a pretty hairy bug.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 3 2009, 14h08
Also, we found and fixed an issue where incorrect albums were being associated with each track. This is quite a lot more serious, and we're sorry this slipped out.

[quote][b][user]jwheare[/user][/b] disse: Also, we found and fixed an issue where incorrect albums were being associated with each track. This is quite a lot more serious, and we're sorry this slipped out.[quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Nov 3 2009, 18h12
I had my fingers crossed y'all might knock out both the playcount and album title bugs at the same time while rummaging beneath the hood. Excellent news!

Thanks for the work and the update!

[quote][b][user]JustSomeOldJoe[/user][/b] disse: I had my fingers crossed y'all might knock out both the playcount and album title bugs at the same time while rummaging beneath the hood. Excellent news! Thanks for the work and the update![quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Nov 4 2009, 20h00
At first glance, this looks a helluva lot better. Zero playcounts are included, and albums appear either correct or not present (which is infinitely better than incorrect).

I'm still seeing a small discrepancy between the number of tracks returned by lib.getTracks (3174) and the number of tracks present in USER->LIB->RECENTLY ADDED (3180). This may just be a case of me adding a few albums (something I don't do) and the albums being listed as an entry, yet no tracks returned (since there really aren't any in the lib as of yet). Or, it could just be attributed to synch latency or something altogether diff. Don't know... but I'll try to spend some time to cross-ref to figure out the exact source since you folks put in the time to implement a fix.

Thanks again!!!

[quote][b][user]JustSomeOldJoe[/user][/b] disse: At first glance, this looks a helluva lot better. Zero playcounts are included, and albums appear either correct or not present (which is infinitely better than incorrect). I'm still seeing a small discrepancy between the number of tracks returned by lib.getTracks (3174) and the number of tracks present in USER->LIB->RECENTLY ADDED (3180). This may just be a case of me adding a few albums (something I don't do) and the albums being listed as an entry, yet no tracks returned (since there really aren't any in the lib as of yet). Or, it could just be attributed to synch latency or something altogether diff. Don't know... but I'll try to spend some time to cross-ref to figure out the exact source since you folks put in the time to implement a fix. Thanks again!!![quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Nov 6 2009, 4h00
lib.getTracks()... still not quite there.
jw,

I ran some tests to confirm everything was pulled back and there are still a few tracks missing. The following two tracks in my lib weren't returned from the library.getTracks() call:

MGMT - Electric Feel
Edvard Grieg - Allegro ma non troppo

Obviously, two tracks ain't no big deal. But, if it's two for me, it could be two hundred for someone else.

Murphy's Law being what it is, if you check it, it'll be difficult to track down and no big deal. If you don't, it'll be the killer bug :)

Not quite accurate yet, but it's definitely close!
[quote][b][user]JustSomeOldJoe[/user][/b] disse: jw, I ran some tests to confirm everything was pulled back and there are still a few tracks missing. The following two tracks in my lib weren't returned from the library.getTracks() call: [ulist][li][url=http://www.last.fm/music/MGMT/_/Electric+Feel]MGMT - Electric Feel[/url][/li][li][url=http://www.last.fm/music/Edvard+Grieg/_/Allegro+ma+non+troppo]Edvard Grieg - Allegro ma non troppo[/url][/li][/ulist] Obviously, two tracks ain't no big deal. But, if it's two for me, it could be two hundred for someone else. Murphy's Law being what it is, if you check it, it'll be difficult to track down and no big deal. If you don't, it'll be the killer bug :) Not quite accurate yet, but it's definitely close![quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 6 2009, 11h52
Ah, it seems tracks that you've only loved and not played or tagged are still not being included. I'll take a look.

[quote][b][user]jwheare[/user][/b] disse: Ah, it seems tracks that you've only loved and not played or tagged are still not being included. I'll take a look.[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 6 2009, 12h01
A little background. For various reasons that I'm not that clued up on, we actually store plays, loves, tags and manual adds in separate datastores at the backend, so querying your library involves joining these sources. These joins are quite cheap, but sometimes we forget about some of the joins required.

To demonstrate this, if we wanted to get a list of any artists that appear in your library anywhere we have to join the following data sources.

* tracks_played
* tracks_loved <- the one we forgot.
* tracks_added
* tracks_tagged
* albums_added
* albums_tagged
* artists_added
* artists_tagged

[quote][b][user]jwheare[/user][/b] disse: A little background. For various reasons that I'm not that clued up on, we actually store plays, loves, tags and manual adds in separate datastores at the backend, so querying your library involves joining these sources. These joins are quite cheap, but sometimes we forget about some of the joins required. To demonstrate this, if we wanted to get a list of any artists that appear in your library anywhere we have to join the following data sources. * tracks_played * tracks_loved <- the one we forgot. * tracks_added * tracks_tagged * albums_added * albums_tagged * artists_added * artists_tagged[quote]

Permalink Citar
- jwheare disse...
- Moderador
- Nov 6 2009, 12h47
I've fixed this in dev but won't go live until the next site release in a couple of weeks.

[quote][b][user]jwheare[/user][/b] disse: I've fixed this in dev but won't go live until the next site release in a couple of weeks.[quote]

Permalink Citar
- JustSomeOldJoe disse...
- Assinante
- Nov 6 2009, 20h15
* tracks_played
* tracks_loved <- the one we forgot.
* tracks_added
* tracks_tagged
* albums_added
* albums_tagged
* artists_added
* artists_taggedCool! I get the reasoning... allows you to manage load horizontally by subject area and scale the same way. Makes sense from a design perspective, it's just a pain in the arse for developers unless the complexity is shielded from 'em at the data abstraction layer!

Assume, based on segmentation convention you outlined, there's also a "tracks_banned" store. The whole "banned and in your lib" vs. "banned and not in your lib" is a semantic oddity to me... so I'll leave it alone :)

Thanks again for the work on resolving this... I'll set my clock to the agile release schedule and give it a run when it gets pushed.

[quote][b][user]JustSomeOldJoe[/user][/b] disse: [quote]* tracks_played * tracks_loved <- the one we forgot. * tracks_added * tracks_tagged * albums_added * albums_tagged * artists_added * artists_tagged[/quote]Cool! I get the reasoning... allows you to manage load horizontally by subject area and scale the same way. Makes sense from a design perspective, it's just a pain in the arse for developers unless the complexity is shielded from 'em at the data abstraction layer! Assume, based on segmentation convention you outlined, there's also a "tracks_banned" store. The whole "banned and in your lib" vs. "banned and not in your lib" is a semantic oddity to me... so I'll leave it alone :) Thanks again for the work on resolving this... I'll set my clock to the agile release schedule and give it a run when it gets pushed.[quote]

Permalink Citar

Usuários anônimos não podem postar mensagens. É preciso fazer login ou criar uma conta para postar nos fóruns.

getTracks... still not quite there

getTracks... still not quite there

this does seem rather silly

lib.getTracks()... still not quite there.