[PLUGIN] SmartQueue

Wheaten · August 2, 2024, 10:13am

Complete library, resulting in a 68MB songs.db

chourmovs_vs · August 2, 2024, 12:04pm

Thinking to this twice, I realized that having a huge database may have an impact on euclydian distance approach
i asked to chatgpt and here is the edifying answer

This phenomenon is completely normal and can occur for several reasons when using distance calculation methods such as Euclidean distance to determine musical similarity in a large database.

Data Concentration: When you have a large database, it is possible that several pieces have very similar characteristics, especially if these characteristics are based on common or very general criteria. This can lead to pieces that are close to each other in terms of Euclidean distance, thus generating results that appear repetitive.

Discretization of Criteria: If the criteria used to measure similarity are themselves general enough or not discriminating enough, the distances between pieces may cluster around certain values. This can reduce the diversity of pieces that appear close to each other.

Clustering Effect: Chunks may naturally group together into clusters in the multidimensional space defined by your criteria. If some clusters are very dense or have a high concentration of chunks, you may get similar results when searching for nearby chunks.

Limitations of Euclidean Distance: Euclidean distance is a similarity measure that may not always capture complex musical nuances, especially if the pieces are not very different according to the chosen criteria. Other similarity measures, such as Manhattan distance or methods based on cosine similarity, may give different and potentially less repeatable results.

"curse effects of dimensionality ", where the distances between points become less significant as the dimension increases, which can make the results more homogeneous and repetitive.

To mitigate these issues, you might try the following approaches:

Normalization and Weighting: Make sure your criteria are normalized and weighted appropriately to avoid any one criterion dominating the others.

Dimensionality Reduction: Use techniques like Principal Component Analysis (PCA) or Independent Component Analysis (ICA) to reduce the dimensionality of the data while retaining the essential information.

Other Similarity Metrics: Try other distance or similarity metrics that might better capture musical nuances, such as cosine similarity or graph-based methods.

Clustering Approaches: Use clustering methods to explore natural groups of songs in your database, which can help you understand and diversify songs that appear to be related.

By adjusting these aspects, you could improve the diversity of the pieces offered while maintaining the relevance of the musical similarity.

Another room of improvement

chourmovs_vs · August 2, 2024, 12:10pm

One thing interesting to test on your large database could be also

blissify playlist --distance cosine 10

instead of default

blissify playlist --distance euclydean 10

There’s also

blissify playlist --seed-song

that change the blissify mechanic

Instead of making a playlist with songs that are only similar to the first song, from the most similar to the least similar (the default), you can make a playlist that queues the closest song to the first song, then the closest song the second song, etc, effectively making “path” through the songs.

I 'm going to test and maybe implement those options

chourmovs_vs · August 2, 2024, 2:55pm

fix smartqueue.prototype.onVolumioStart

Wheaten · August 2, 2024, 5:28pm

Currently at my holiday residence. Only have 256GB drive here. Will be after the weekend.

chourmovs_vs · August 2, 2024, 5:40pm

No problem, I go to Paris 2024 tomorrow for 1 week
Hope I’ll be back one day (French Joke)

Wheaten · August 2, 2024, 5:44pm

As Jeremy Clarkson stated:
The only good thing in Paris, is the road to Spain…

chourmovs_vs · August 2, 2024, 6:28pm

version 1.1.5

add Cosine and Seed option

Seems to Fix repetition issue here (especially Seed On)

Time for me to say good-bye

Wheaten · August 10, 2024, 10:46pm

Installed on a factory resetted rPi4, on Volumio V3.742.

Error executing corechooser.py: Command failed: /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py
kill: (1106): Operation not permitted
Terminated

chourmovs_vs · August 11, 2024, 6:19am

Thx Wheaten, I will investigate later today
If you have any further log it could help

Wheaten · August 11, 2024, 7:57am

no further log. copied it from live log

chourmovs_vs · August 11, 2024, 8:13am

output of the command /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py I meant

Wheaten · August 11, 2024, 8:47am

volumio@rpi4-argon:~/smartqueue$  /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py
Loading configuration from /data/configuration/user_interface/smartqueue/config.json
Configuration loaded: {'Autoqueue': {'type': 'boolean', 'value': False}, 'Blissmixer': {'type': 'boolean', 'value': True}, 'Tracksn': {'type': 'number', 'value': 10}, 'Drift': {'type': 'number', 'value': 1}, 'Cosine': {'type': 'boolean', 'value': True}, 'Seed': {'type': 'boolean', 'value': True}, 'address': {'type': 'string', 'value': '192.168.1.111'}}
Running core chooser under conditions...
Raw values: Autoqueue: False, Blissmixer: True
Autoqueue: False, Blissmixer: True
Running command for Blissmixer true: /usr/bin/pgrep python3 | xargs -r /bin/kill -15 && /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py
kill: (1130): Operation not permitted
Terminated

It only runs when i do it as sudo:
sudo /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py

seems the issue is with:
/usr/bin/pgrep python3 | xargs -r /bin/kill -15

Which seems already a no-go as your killing all running python pids. You are only entitled to kill the ones owned by smartqueue.
For example:

volumio@rpi4-argon:~$ ps -axo pid,cmd | egrep '/smartqueue/.*py'
 4139 /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py
 4140 /bin/sh -c /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py
 4141 /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py

volumio@rpi4-argon:~$ pgrep -f /smartqueue.*py
4139
4140
4141

as last I see multiple identical processes running of ‘installbliss.sh’ and ‘updatebliss’:

volumio@rpi4-argon:~$ ps -axo pid,cmd | egrep '/smartqueue/unit/'
 2186 shellinaboxd -t -b -p 10002 --no-beep -s /install/:volumio:volumio:/:/bin/bash /data/plugins/user_interface/smartqueue/unit/installbliss.sh
 2187 shellinaboxd -t -b -p 10002 --no-beep -s /install/:volumio:volumio:/:/bin/bash /data/plugins/user_interface/smartqueue/unit/installbliss.sh
 2189 shellinaboxd -t -b -p 10003 --no-beep -s /update/:volumio:volumio:/:/bin/bash /data/plugins/user_interface/smartqueue/unit/updatebliss.sh
 2190 shellinaboxd -t -b -p 10003 --no-beep -s /update/:volumio:volumio:/:/bin/bash /data/plugins/user_interface/smartqueue/unit/updatebliss.sh
 4139 /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/corechooser.py
 4140 /bin/sh -c /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py
 4141 /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py

volumio@rpi4-argon:~$ pgrep -f /.*bliss.*sh
2186
2187
2189
2190

chourmovs_vs · August 11, 2024, 9:59am

You’re right again

I need to review my “kill logic” to focus on Smarqueue instance otherwise it could break other plugins
I may run processes as PID 1000 / UID 1000 or sudo

I’m going to work on this
BEDANKT

Wheaten · August 11, 2024, 10:18am

It still needs a lot of testing, before you release it again.
Forced the command ‘’ but then the next error appears.

volumio@rpi4-argon:~$ /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py
Checking current URI and last queue URI...
Current URI: mnt/NAS/QNAP/Sam Jr/(2022) - Sam Jr/0106 - Sam Jr. - World Bangin' On My Door.flac
Last Queue URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Comparing current URI with last queue URI...
Not on the last track.
Checking current URI and last queue URI...
Current URI: mnt/NAS/QNAP/Sam Jr/(2022) - Sam Jr/0106 - Sam Jr. - World Bangin' On My Door.flac
Last Queue URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Comparing current URI with last queue URI...
Not on the last track.
Checking current URI and last queue URI...
Current URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Last Queue URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Comparing current URI with last queue URI...
Currently on the last track, executing scripts bliss.py...
Tracksn variable: 10 (type: <class 'int'>)
Current track artist: Golden Earring
Current track title: Daddy buy me a girl
Current track genre: Rock
Drift variable: 1 (type: <class 'int'>)
Configuration loaded: {'Autoqueue': {'type': 'boolean', 'value': False}, 'Blissmixer': {'type': 'boolean', 'value': True}, 'Tracksn': {'type': 'number', 'value': 10}, 'Drift': {'type': 'number', 'value': 1}, 'Cosine': {'type': 'boolean', 'value': True}, 'Seed': {'type': 'boolean', 'value': True}, 'address': {'type': 'string', 'value': '192.168.1.111'}}
Seed:: True, Cosine: True
Running blissify to generate playlist...
^CTraceback (most recent call last):
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 226, in <module>
    main()
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 217, in main
    related_tracks = run_blissify_and_export_playlist(config,tracksn)
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 84, in run_blissify_and_export_playlist
    capture_output=True, text=True
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1681, in _communicate
    ready = selector.select(timeout)
  File "/usr/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
Traceback (most recent call last):
  File "/data/plugins/user_interface/smartqueue/unit/coreb.py", line 61, in <module>
    main()
  File "/data/plugins/user_interface/smartqueue/unit/coreb.py", line 52, in main
    subprocess.run(["/usr/bin/python3", "/data/plugins/user_interface/smartqueue/unit/bliss.py"], check=True)
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 931, in communicate
    self.wait()
  File "/usr/lib/python3.7/subprocess.py", line 990, in wait
    return self._wait(timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1624, in _wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib/python3.7/subprocess.py", line 1582, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
volumio@rpi4-argon:~$ pgrep -f /.*bliss.*sh
volumio@rpi4-argon:~$ ps -axo pid,cmd | egrep '/smartqueue/unit/'
 7329 grep -E /smartqueue/unit/
volumio@rpi4-argon:~$ /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/coreb.py
Checking current URI and last queue URI...
Current URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Last Queue URI: mnt/NAS/QNAP/Golden Earring/(2000) - The Devil Made Us Do It - 35 Years/0104 - Golden Earring - Daddy buy me a girl.flac
Comparing current URI with last queue URI...
Currently on the last track, executing scripts bliss.py...
Tracksn variable: 10 (type: <class 'int'>)
Current track artist: Golden Earring
Current track title: Daddy buy me a girl
Current track genre: Rock
Drift variable: 1 (type: <class 'int'>)
Configuration loaded: {'Autoqueue': {'type': 'boolean', 'value': False}, 'Blissmixer': {'type': 'boolean', 'value': True}, 'Tracksn': {'type': 'number', 'value': 10}, 'Drift': {'type': 'number', 'value': 1}, 'Cosine': {'type': 'boolean', 'value': True}, 'Seed': {'type': 'boolean', 'value': True}, 'address': {'type': 'string', 'value': '192.168.1.111'}}
Seed:: True, Cosine: True
Running blissify to generate playlist...
Traceback (most recent call last):
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 226, in <module>
    main()
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 217, in main
    related_tracks = run_blissify_and_export_playlist(config,tracksn)
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 116, in run_blissify_and_export_playlist
    with open("/home/volumio/bliss.m3u", "w", encoding='utf-8') as f:
PermissionError: [Errno 13] Permission denied: '/home/volumio/bliss.m3u'
Traceback (most recent call last):
  File "/data/plugins/user_interface/smartqueue/unit/coreb.py", line 61, in <module>
    main()
  File "/data/plugins/user_interface/smartqueue/unit/coreb.py", line 52, in main
    subprocess.run(["/usr/bin/python3", "/data/plugins/user_interface/smartqueue/unit/bliss.py"], check=True)
  File "/usr/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '/data/plugins/user_interface/smartqueue/unit/bliss.py']' returned non-zero exit status 1.
volumio@rpi4-argon:~$ /usr/bin/python3 /data/plugins/user_interface/smartqueue/unit/bliss.py
Tracksn variable: 10 (type: <class 'int'>)
Current track artist:
Current track title:
Current track genre:
Drift variable: 1 (type: <class 'int'>)
Configuration loaded: {'Autoqueue': {'type': 'boolean', 'value': False}, 'Blissmixer': {'type': 'boolean', 'value': True}, 'Tracksn': {'type': 'number', 'value': 10}, 'Drift': {'type': 'number', 'value': 1}, 'Cosine': {'type': 'boolean', 'value': True}, 'Seed': {'type': 'boolean', 'value': True}, 'address': {'type': 'string', 'value': '192.168.1.111'}}
Seed:: True, Cosine: True
Running blissify to generate playlist...
Traceback (most recent call last):
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 226, in <module>
    main()
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 217, in main
    related_tracks = run_blissify_and_export_playlist(config,tracksn)
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 116, in run_blissify_and_export_playlist
    with open("/home/volumio/bliss.m3u", "w", encoding='utf-8') as f:
PermissionError: [Errno 13] Permission denied: '/home/volumio/bliss.m3u'
volumio@rpi4-argon:~$ subprocess.CalledProcessError: Command '['/usr/bin/python3', '/data/plugins/user_interface/smartqueue/unit/bliss.py']' returned non-zero exit status 1.
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 217, in main
    related_tracks = run_blissify_and_export_playlist(config,tracksn)
  File "/data/plugins/user_interface/smartqueue/unit/bliss.py", line 116, in run_blissify_and_export_playlist
    with open("/home/volumio/bliss.m3u", "w", encoding='utf-8') as f:
PermissionError: [Errno 13] Permission denied: '/home/volumio/bliss.m3u'
-bash: subprocess.CalledProcessError:: command not found

chourmovs_vs · August 11, 2024, 1:10pm

My additional testing deserve nothing, it works perfectly on my 3 machines (arm7l x2 and virtual x86) on fresh reseted 3.742
Maybe I’m missing an rpi4 at home to test further, but the behaviour on your machine resist to my understanding, i feel a little bit stalled…

Wheaten · August 11, 2024, 1:40pm

I am only reporting what’s happening on my rPi4, using this as it’s the most used device with Volumio.
Started from a fresh install, did my library scan, downloaded the plugin:
git clone https://github.com/chourmovs/volumio-plugins-sources.git

And when using it, it runs into errors, can’t make more of it.

chourmovs_vs · August 11, 2024, 3:35pm

Just one stupid question
did you download binary after Installed on a factory resetted rPi4 ?

I reproduced exactly your bugs on one of my machine by forgetting to download binary
finger crossed

Wheaten · August 11, 2024, 5:31pm

Clean install, so I did download the binary.
Ran 2 tests:

Factory reset, installed plugin and bliss. Only took a copy of music.db to save 8 hours.
Fresh install, installed plugin and bliss. Again took a copy of music.db to save 8 hours.

chourmovs_vs · August 12, 2024, 7:36am

Hi Weathen

For Arm7l arch, the binary I put on my site for download is not the one you prepared me as below but one compiled on my primo ( I believed it was ok because it worked on my RPI3)

Could you try to replace the binary in /home/volumio/.cargo/bin by you rpi4 generated one

I believe this won’t fix your issue (which looks like a permission issue following your log) but this could be interesting to test to “close doors”

Otherwise I’ve already fixed this :

But i don’t really see why i would have to work on this :

Thx by advance