Thursday 13 June 2013

MEASUREMENTS: Part II: Bit-Perfect Audiophile Music Players - JPLAY (Windows).

There's something happening here,What it is ain't exactly clear.-- Buffalo Springfield

Welcome to Part II of the "Bit-Perfect" roundup for Windows.

In this installment, I'll focus on JPLAY (most recent version 5.1) which has been highly promoted around many of the audiophile web sites like 6moons, EnjoyTheMusic ("First Step to Heaven = JPLAY"!), High Fidelity (Polish), TNT-Audio, recently AudioStream plus all the hoopla around the terribly misleading series of articles on computer audio by Zeilig & Clawson in "The Absolute Sound" in early 2012.

If you browse around the JPLAY website (or hang out on various computer audiophile forums), you run into suggestions that process priority, memory playback, task switching latency, driver type (eg. ASIO, WASAPI, Kernel Streaming) all have some kind of effect on the playback audio quality (usual digital whipping boy for bad sound quality is of course the dreaded jitter). Sure, computers can be complicated, but is it possible for these factors to affect audio output significantly enough to hear and common enough to worry about with a modern USB DAC like in my test set-up? Maybe, but like many subtle things in life "reported" to be true, it's hard to know with confidence unless systematically tested.  To address these "issues" JPLAY even has a mode where the screen gets turned off, OS processes are halted, drivers turned off, etc. the famed "hibernate mode". Since the keyboard is turned off, the computer comes back to life after the music has completed playing or if you remove a USB stick during playback! Realize that this makes the computer even less interactive than a disc spinner!

Mitchco has already used the DiffMaker program to compare JPLAY and JRiver about a year ago but was not able to detect a difference. Likewise, there have been some heated discussions over on Hydrogen Audio regarding this program.

Well, let us put this program and various of its settings on the test bench and see what comes up...

Firstly, the setup - same as Part I:
ASUS Taichi (*running JPLAY/foobar*) Win8 x64 --> shielded USB (Belkin Gold) --> TEAC UD-501 DAC --> shielded 6' RCA --> E-MU 0404USB --> shielded USB --> Win8 Acer laptop



Remember that JPLAY is essentially a "playback engine" using its own buffering algorithm and runs as an Audio Service under Windows. In this regard, it is quite unique. By itself, it has a very bare-bones text-based interface. Therefore, for ease of testing, I'm going to be using JPLAY through foobar2000 as an ASIO output "device". Within JPLAY settings, one can then specify the actual audio device and which driver to use (eg. ASIO, WASAPI, Kernel Streaming...), along with specific parameters like buffer size. There are four "engines" - Beach, River, Xtream (Kernel Streaming only), and ULTRAstream (Kernel Streaming & Windows 8 only) - I am unaware of how they are supposed to differ. Note that for all these tests, I had "Throttle" turned ON which is supposed to increase priority to JPLAY (and hence diminish task priority of other processes). Volume control was turned OFF. According to the manual, there are even "advanced settings" called DriverBuffer, UltraSize, and XtreamSize which I did not bother playing with since I presume they're deemed less important if one has to go into the Registry to adjust. Thankfully, the trial version will play ~2 minutes of uninterrupted audio before inserting ~5 seconds of silence. This is enough time to measure the audio quality using my standard tests.

I. RightMark 6.2.5 (PCM 16/44 & 24/96)

Let us start with JPLAY using the ASIO & WASAPI drivers. I think it is important to remember that the makers of JPLAY recommend using Kernel Streaming instead. My suspicion is that JPLAY essentially just passes information off to these ASIO and WASAPI drivers so logically there would be little reason to believe measurements & sound quality should change at all. Furthermore, the CPU utilization for the process "jplay.exe" with ASIO and WASAPI remained low during playback - on the order of <2% peak and usually <1% using the i5 processor. As we see later, this dramatically changes with Kernel Streaming where the program can actually get closer to the hardware and do direct processing in "kernel mode".

Here's the data for JPLAY at 16/44 with ASIO and the two "engines" that support ASIO - "Beach" and "River". When you see a number like "ASIO 2", that refers to the buffer setting (number of samples). The software recommends using the smallest buffer size with "DirectLink" being 1 sample. I was not able to get the DirectLink setting working without occasional blips and errors once I started doing Kernel Streaming, but 2 works well for 16/44, 4 was good for 24/96 so I standardized on those settings throughout. As you can see, I also measured with buffer of 64 samples to see if that made any difference (I see from the manual that Xtreme only works for buffer sizes up to 32 samples - it didn't complain when I set it to 64):


Note that I used the standard foobar2000 + ASIO audio output as the comparison measurement in the leftmost column.

Frequency Response:


Noise:


THD:


Stereo Crosstalk:


For some reason, WASAPI refused to initiate for me at 16/44. But it did work at 24/96, so I have included that here as well. Again, foobar + ASIO was used for comparison on the left:

Frequency Response:

Noise:

THD:


Stereo Crosstalk:

So far, no real difference between the JPLAY playback and foobar2000.

Now, lets deal with Kernel Streaming. It is with KS that we see significant CPU utilization by "jplay.exe". Instead of <2% peak CPU utilization above, with low buffers like a setting of 2 samples for 16/44, I see peak CPU utilization of 13%, and with a 4-sample buffer at 24/96, peak CPU jumps to 16% with my laptop's i5 and the "Xtream" engine. The "ULTRAstream" engine chews up even more CPU cycles by another 1-2% above those previous numbers. The moment the buffer size was increased to 64 samples, CPU utilization dropped to 3% with 16/44 and 6% with 24/96 peak. It looks like only when Kernel Streaming is used, does JPLAY actually kick in to maintain those tiny buffer settings.

Starting with 16/44, here's Kernel Streaming with mainly the "Xtream" and "ULTRAstream" engines since these two cannot be used with ASIO. I included "Beach" in these measurements as well out of interest. The test on the far right with "hiber" refers to the use of the "hibernate mode" where the computer screen, OS processes, drives, etc. turn off during playback. Again, foobar + ASIO was used as the comparison:

Frequency Response:

Noise:

THD:

Stereo Crosstalk:

Here is 24/96:

Frequency Response:

Noise:

THD:

Stereo Crosstalk:

So far, despite all the changes in CPU utilization, different audio "engines" and buffer settings, I see no substantial change in the measurements that would suggest a qualitative difference in terms of the analogue output signal from my DAC.

JPLAY supposedly can support DSD playback. I did not test this function.

Part II: Dunn J-Test for jitter

I did a whole suite of J-Test with all the different audio "engines", either 2 or 64 sample buffers, ASIO/WASAPI/KS, also tried the "hibernate" mode. For brevity, here's a selection of 16-bit jitter spectra using various settings:

foobar2000 ASIO:


Beach ASIO 64-sample buffer:

River ASIO 2-sample buffer:

Xtream Kernel Streaming 64-sample buffer:

ULTRAstream Kernel Streaming 2-sample buffer:

ULTRAstream Kernel Streaming + Hibernate 2-sample buffer:

Essentially no difference in the J-Test spectra.

Recall that the 24-bit Dunn J-Test is done with a 24/48 signal. While doing this test using Kernel Streaming mode, something strange was found.

This is what the 24-bit Dunn J-Test looks like with foobar2000 + ASIO (notice primary signal at 12kHz rather than 11kHz with the 16-bit test):
The 24-bit LSB modulation signal is buried under the noise level. This is quite normal.

Here is Beach + ASIO:

Again, quite normal - we're still using the TEAC ASIO driver.

Look what happens with 24-bit J-Test Beach + Kernel Streaming (doesn't matter what buffer size):

Eh? What's with all the modulation signal bursting through like with the 16-bit test?!

My suspicion is that JPLAY isn't handling the last 8-bits in the 24-bit data properly... One possible scenario is where the last 8 bits got flipped from LSB to MSB, thus causing the LSB signal to show through at a higher level. With RightMark, this is what it looks like:


We've "lost" the last 7 bits of resolution at 24/48 with Kernel Streaming causing the dynamic range to drop to 102dB (17-bits resolution) instead of the usual >110dB (24-bits into the noise floor). I considered the possibility that this may represent some sort of dithering but why would it be applied to 24/48 and not 24/96?

Strangely, this anomaly did not show up at 24/96. I did not bother to check if 88/176/192kHz sampling rates were affected.

Part III: DMAC Protocol

Time for the machine listening test of 24/44 composite music using Audio DiffMaker. As shown in previous blog posts, this test seems to be quite good in detecting even relatively small changes like minute EQ adjustments, difference between AAC, MP3, etc... This test is also similar to what mitchco did last year.

Here are measurements of a few settings in JPLAY in comparison to foobar ASIO as reference. I included the most "extreme" one that I could perform - ULTRAstream + 2-sample buffer + hibernate mode. As usual a couple of "sensitivity tests" with MP3 320kbps and slight EQ change in foobar for comparison:

The machine was able to correlate the null depth down to around 90dB with all the JPLAY settings and foobar suggesting no significant difference in sound in comparison to the reference foobar + ASIO playback.

Part IV: Conclusion

Based on these measurements, a few observations:

1. 24/48 with Kernel Streaming appears buggy as shown with the 24-bit J-Test and RightMark result. Don't use it (as of version 5.1) if you want accurate sound. However, subjectively it still sounds OK to me since it's still accurate down to 16-bits at least. ASIO works fine. 24/96 is fine. I don't know if other sampling rates with 24-bit data are affected. For some reason I could not get WASAPI 16/44 to work with JPLAY even though it was fine with foobar2000.

2. Technically, JPLAY appears to be bit perfect with 16/44 and likely 24/96 based on my tests (of course we cannot say this for 24/48). Since the program claims to be bit perfect, this is good I guess.

3. I was unable to detect any evidence of sonic difference at 16/44 and 24/96 compared to a standard foobar set-up. RightMark tests look essentially the same. Over the months of testing, I see no evidence still that software changes the jitter severity with CPU load, different software, even different DACs (as I had postulated awhile back in this post). DiffMaker null tests were also unable to detect any significant difference in the "sound" of the analogue output from the TEAC UD-501.

4. Still no evidence that extreme settings like "hibernate mode" which reduces the utility of the computer makes any sonic difference. Of course, it's possible that this could make a difference with some very slow machines like a 1st generation single-core Atom processor with small buffer settings doing Kernel Streaming... But in that case, why not just increase the buffer size with Kernel Streaming (why all the "need" for low buffer settings and obsession over low latency for just playback?!) or just go with an efficient ASIO/WASAPI driver? I'd also recommend a processor upgrade if you're still rocking an old Atom!

Over the 3 nights I was performing these tests, I took time to listen to music with the various JPLAY settings - probably ~3 hours total. It sounds good subjectively (other than the brief interruptions every 2 minutes or so for the trial version). The i5 computer shows a little bit of lag with Kernel Streaming and low buffer size if I'm trying to do something else. I listened to Donald Fagen's The Nightfly and Peter Gabriel's So (2012 remaster) in 24/48 with Kernel Streaming and didn't think they sounded bad despite the issue I found (remember, the anomaly was down below 16-bits). Dire Strait's Brother In Arms sounded dynamic and detailed as usual. Likewise Richard Hickox & LSO's Orff: Carmina Burana (2008, 24/88 SACD rip) sounds just as ominous. (Almost) Instantaneous A-B'ing is possible in foobar changing ASIO output from the TEAC driver to JPLAY and I did not notice any significant subjective tonal, amplitude, or resolution difference using my Sennheiser HD800 headphones with the TEAC UD-501 DAC flipping back and forth a number of times.

Bottom line: With a reasonably standard set-up as described, using a current-generation (2013) asynchronous USB DAC, there appears to be no benefit with the use of JPLAY over any of the standard bit-perfect Windows players tested previously in terms of measured sonic output. Nor could I say that subjectively I heard a difference through the headphones. If anything, one is subjected to potential bugs like the 24/48 issue (I didn't run into any system instability thankfully), and the recommended Kernel Streaming mode utilizes significant CPU resources when buffer size is reduced (which the software recommends doing). I imagine that CPU utilization would be even higher if I could have activated the DirectLink (1-sample buffer) setting.

Finally, there's the fact that this program costs €99. A bit steep ain't it? JRiver costs US$50, Decibel on the Mac around $35, foobar2000 FREE and these all feature graphical user interfaces and playlists at least! Considering my findings, I'm unclear with what DAC or computer system one would find tangible benefits after spending €99 for this program.

As usual, I welcome feedback especially with objective data or controlled test results (any JPLAY software developers care to comment on how the audio engines were "tuned"?). I would also welcome independent testing to see if my findings can be verified on other hardware combinations (especially that 24/48 issue).

Music selection tonight: Paavo Järvi & Orchestre de Paris - Fauré: Messe de Requiem (2011). Lovely rendition of Pavane for mixed choir!

As usual... Enjoy the tunes... :-)

12 comments:

  1. For a fun exercise lets sit on a chair in the subjective corner and think of some random 'counter arguments' why the tests are flawed.

    A: NO quantifiable audible differences are found based on jitter tests, RMAA and a null test, who cares it is common knowledge RMAA and null tests cannot show what the ears and the brain CAN detect easily.
    B: Every person/ hearing is different and we can obviously hear things others can't.
    C: It's another one of those ignorant EE's with their measurement equipment with bad hearing out to show us how wrong we are, they are to be pitied.
    D: The 'objectivsists' are biased because they (the measuring kind) disregard the audible differences as they are 'impossible' and thus they don't hear it even when it stares them in the face.
    E: the used equipment is (far) below par and simply doesn't have the resolution, speed, FR needed to show the obvious differences.
    F: They aren't trained to listen out for the 'effects', they can only stare at their read-out/screens
    G: hearing is complicated and consists of more than just FR and distortion and the important factors are not known (yet) as science is still in it's infancy while human hearing has been around and perfected through ages.
    H: Evaluation of complex sounds cannot be done by a computer as the modeling of the ear is not included, the hearing is infinitely better at this.
    I: I can hear differences when swapping (fill in: cables, mains-cables, rectifiers, resistors, capacitors, e.t.c.) while no scientific tests have ever been able to show this yet I and (pick your person) have verified this or that over and over again reliably so why would this test be conclusive.
    J: correlation of the DMAC test is 90dB at most which indicates that test is flawed as the subtleties in music are below -110dB which I can easily detect as ( fill in: placement of instruments, feel of reality e.t.c.)
    K: The MANY confirmations about differences (fill in all audio disciplines) coming in independently from all over the world cannot be ignored and this is MORE than enough to prove it.
    L: The test is so stupid I won't even bother to reply to those that have already made up their mind and aren't open to anything.

    Did I get most of them right ?

    To me personally the tests are valid and prove what is 'logical'.
    I am, however, one of those people with bad or untrained hearing, below par equipment or am stuck in my 'science is important' delusion.
    Happy with that and the fact that some take the trouble to do these tests. so thanks...

    I hope software designers DO chip in with their secrets and HOW they determined the improve in SQ but believe they make a living out of 'the hype' and the secrecy/know how.

    ReplyDelete
    Replies
    1. :-)

      I think you pretty well got 'em all! Good analysis.



      Delete
  2. Thanks for a very enlightening test.
    Any chance you could include the latest XBMC (12.2) in your audio bit-perfect test? While I cannot hear any diff between Foobar and JPlay there is a pretty audible one between Foobar & XBMC (all using the same WASAPI output). XBMC sounds a bit like someone put a veil on the speakers. Not a huge diff but clearly audible. And I wonder if that would show up in your test setup.

    ReplyDelete
  3. I needs to spend some time learning more or understanding more.Thanks for excellent information I was looking for this information.


    Home CCTV camera & Ford falcon DVD player

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. The answer is time correct music playing, this is how music make feelings. Show me how you would like to measure this. There is a problem in your test. I can't read feelings. Is that because you didn't measure it. Am I right? That is how you write about it. This test is much to less, that what would discribe the whole music thing. ;)

    (sorry for my english)

    ReplyDelete
    Replies
    1. You are ascribing "feelings" to something the hardware or software evokes in you rather than where I believe it originates - the artist and his/her abilities within the music you love. All the hardware / software can do is be faithful to reproducing the waveforms fed into them. I'm showing that JPlay is faithful in this respect with my DAC, *but nothing more than any other bit-perfect software*. Unless you can describe another test, I'm sorry, but it's not good enough to claim there's something more... If there is more, then it's the onus of people like JPlay who claim a difference to *show* using whatever means they believe necessary.

      Time correction?! When there is no issue in tests like the J-Test for jitter, this makes it very unlikely there is an issue at all with time fluctuations at the level of the analogue output. If you imply interchannel imbalance with time (eg. stereo channels shifting in time), then you better look at the DAC because something's wrong! Software will not make this better.

      Delete
  6. I suppose I've selected an unbelievable and interesting blog.
    how to get more plays on Soundcloud

    ReplyDelete
  7. >24/48 with Kernel Streaming appears buggy as shown with the 24-bit J-Test and RightMark result<

    It seems the author didn't read the JPlay manual.

    When Bitsteam is set to 'Native', 44.1&48kHz sample rates are always played in 16bit, that's what the JPlay manual says.
    When feeding 44.1&48kHz sample rates in 24bit, 8bit will be truncaded by JPlay, no dither in JPlay, that's what the author found.
    For the Bit-perfect playback of 24bit files at all sample rates, the Bitstream setting should be 'Force 24bit'.

    ReplyDelete
  8. maybe your teac-Dac is your bottleneck in your set-up, or your amp your speakers, your room?
    the dac, the amp, the speaker and your room are the main building blocks for audiophile listening pleasure.
    the last step will be the software player. if the main nbuilding blocks are not offering superior performance
    you will not hear any difference between different software players...

    maybe your setup is unable to cover the differences? i guess.
    to me its evident, this jplay with foobar as frontend sound so much better than any other configurations within foobar or other players i've used.

    off course i'm using the phenomenal L.K.S Audio MH-DA003 ES9018 dac, my exceptional 3a audio design midi master speakers and the big old denon pma 1315r amp.
    adding and in my view the most important step in this setup is a room eq like the wonderful mathaudio room eq. investing in a BEYERDYNAMIC MM1 mic is a smart move.
    to make the story short: in this setup the differences between foobar with jplay to foobar with all other possibilities, hqplayer or other players et cetera is huge!!

    measurements say nothing when it is about listening to music in a real setup.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. So, I'm a measurement type, but there's apparently some measurement(s) missing that DO affect perceived sound. I was skeptical, but tried JPlay. There are UNDOUBTEDLY differences in the sound. Whether they are 'good' or not is a whole different story. There are some settings that create a larger soundstage and further vocals and others produce a very small soundstage. Some are just 'harsher' sounding and others are 'smoother', but as of writing this I can't find a setting that is better all around than an output from Tidal directly to the WASAPI driver. That said, there are substantial differences in sound across the JPlay settings - not the kind you have to try convince yourself of. I wish I knew what these differences in sound could be attributed to..

    ReplyDelete