Skybuck's Racing Random Number Generator V3 (algorithm fix..

Well something is wrong with the program or the theory.

I run ent.exe in a dos box on Windows XP x64 Pro Edition.

I tried two files:

1. 212 MB AVI (video only)

Entropy: 6.211472 bits per byte.

Reduction would be 22 percent.

The same file compressed with WinRar:

79.2 MB, that's clearly more than 50% compression.

2. 900 KB BMP (Picture)

Reduction would be something like 15 procent.

It was way over 50 procent

WinRar file only 300 KB or so.

Try it yourself I would say

Bye,
Skybuck.

"William" <william_156.TakeThisOut@gmail.com> wrote in message
news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
> No. Your dead wrong there. Have you even tried this?
>
> Optimum compression means the theoretical maximum you could (losslessy)
> compress a file.
>
> No program could compress file a file filled with random bytes. If it
> could then by definition the data would not be random.
>
> That's how compression works - by finding patterns!
>
> Skybuck Flying wrote:
>> What is ment with optimum compression ?
>>
>> Whatever the case may be, the program is useless to determine what the
>> maximum compressed file size would be.
>>
>> WinZip and WinRar for example compress much better than ent.exe indicates
>> should be theoretically possible.
>>
>> Bye,
>> Skybuck.
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Simply put I don't believe you. Can you please upload a sample avi and
compressed archive to a service like rapidshare.

Here is a 10k file filled with random noise from delphis random number
generator:

http://rapidshare.de/files/38183629/rand.bin.html

Try and compress it , can't be done!

Files with high entropy can't be compressed for the same reason you when
you zip a zip file it won't get smaller. Think about it.



Skybuck Flying wrote:
> Well something is wrong with the program or the theory.
>
> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>
> I tried two files:
>
> 1. 212 MB AVI (video only)
>
> Entropy: 6.211472 bits per byte.
>
> Reduction would be 22 percent.
>
> The same file compressed with WinRar:
>
> 79.2 MB, that's clearly more than 50% compression.
>
> 2. 900 KB BMP (Picture)
>
> Reduction would be something like 15 procent.
>
> It was way over 50 procent
>
> WinRar file only 300 KB or so.
>
> Try it yourself I would say
>
> Bye,
> Skybuck.
>
> "William" <william_156.TakeThisOut@gmail.com> wrote in message
> news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
>> No. Your dead wrong there. Have you even tried this?
>>
>> Optimum compression means the theoretical maximum you could (losslessy)
>> compress a file.
>>
>> No program could compress file a file filled with random bytes. If it
>> could then by definition the data would not be random.
>>
>> That's how compression works - by finding patterns!
>>
>> Skybuck Flying wrote:
>>> What is ment with optimum compression ?
>>>
>>> Whatever the case may be, the program is useless to determine what the
>>> maximum compressed file size would be.
>>>
>>> WinZip and WinRar for example compress much better than ent.exe indicates
>>> should be theoretically possible.
>>>
>>> Bye,
>>> Skybuck.
>
>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

I have my own webdrive, I put a BMP on it for you, and the compressed RAR
file and the TEXTENT.TXT which contains the output of the "ent.exe TEST.BMP"

http://members.home.nl/hbthouppermans/EntropyTest/

Test.bmp is 900 KB
Test.rar is 347 KB

"Ent.exe Test.bmp" output:

Entropy = 6.723972 bits per byte.

Optimum compression would reduce the size
of this 921654 byte file by 15 percent.

Chi square distribution for 921654 samples is 2744487.23, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 40.3445 (127.5 = random).
Monte Carlo value for Pi is 3.819698065 (error 21.58 percent).
Serial correlation coefficient is 0.843178 (totally uncorrelated = 0.0).

15 procent of 900 KB = 135 KB

Clearly WinRar compressess better than that.

Bye,
Skybuck.

"William" <william_156.RemoveThis@gmail.com> wrote in message
news:EpSdnSS4O9HFbefa4p2dnAA@giganews.com...
> Simply put I don't believe you. Can you please upload a sample avi and
> compressed archive to a service like rapidshare.
>
> Here is a 10k file filled with random noise from delphis random number
> generator:
>
> http://rapidshare.de/files/38183629/rand.bin.html
>
> Try and compress it , can't be done!
>
> Files with high entropy can't be compressed for the same reason you when
> you zip a zip file it won't get smaller. Think about it.
>
>
>
> Skybuck Flying wrote:
>> Well something is wrong with the program or the theory.
>>
>> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>>
>> I tried two files:
>>
>> 1. 212 MB AVI (video only)
>>
>> Entropy: 6.211472 bits per byte.
>>
>> Reduction would be 22 percent.
>>
>> The same file compressed with WinRar:
>>
>> 79.2 MB, that's clearly more than 50% compression.
>>
>> 2. 900 KB BMP (Picture)
>>
>> Reduction would be something like 15 procent.
>>
>> It was way over 50 procent
>>
>> WinRar file only 300 KB or so.
>>
>> Try it yourself I would say
>>
>> Bye,
>> Skybuck.
>>
>> "William" <william_156.RemoveThis@gmail.com> wrote in message
>> news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
>>> No. Your dead wrong there. Have you even tried this?
>>>
>>> Optimum compression means the theoretical maximum you could (losslessy)
>>> compress a file.
>>>
>>> No program could compress file a file filled with random bytes. If it
>>> could then by definition the data would not be random.
>>>
>>> That's how compression works - by finding patterns!
>>>
>>> Skybuck Flying wrote:
>>>> What is ment with optimum compression ?
>>>>
>>>> Whatever the case may be, the program is useless to determine what the
>>>> maximum compressed file size would be.
>>>>
>>>> WinZip and WinRar for example compress much better than ent.exe
>>>> indicates should be theoretically possible.
>>>>
>>>> Bye,
>>>> Skybuck.
>>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

I stand corrected. This goes against what I thought I knew about
compression!

Wikipedia has an article on the subject. But the maths is way over my head.

http://en.wikipedia.org/wiki/Information_entropy

Skybuck Flying wrote:
> I have my own webdrive, I put a BMP on it for you, and the compressed RAR
> file and the TEXTENT.TXT which contains the output of the "ent.exe TEST.BMP"
>
> http://members.home.nl/hbthouppermans/EntropyTest/
>
> Test.bmp is 900 KB
> Test.rar is 347 KB
>
> "Ent.exe Test.bmp" output:
>
> Entropy = 6.723972 bits per byte.
>
> Optimum compression would reduce the size
> of this 921654 byte file by 15 percent.
>
> Chi square distribution for 921654 samples is 2744487.23, and randomly
> would exceed this value 0.01 percent of the times.
>
> Arithmetic mean value of data bytes is 40.3445 (127.5 = random).
> Monte Carlo value for Pi is 3.819698065 (error 21.58 percent).
> Serial correlation coefficient is 0.843178 (totally uncorrelated = 0.0).
>
> 15 procent of 900 KB = 135 KB
>
> Clearly WinRar compressess better than that.
>
> Bye,
> Skybuck.
>
> "William" <william_156 RemoveThis @gmail.com> wrote in message
> news:EpSdnSS4O9HFbefa4p2dnAA@giganews.com...
>> Simply put I don't believe you. Can you please upload a sample avi and
>> compressed archive to a service like rapidshare.
>>
>> Here is a 10k file filled with random noise from delphis random number
>> generator:
>>
>> http://rapidshare.de/files/38183629/rand.bin.html
>>
>> Try and compress it , can't be done!
>>
>> Files with high entropy can't be compressed for the same reason you when
>> you zip a zip file it won't get smaller. Think about it.
>>
>>
>>
>> Skybuck Flying wrote:
>>> Well something is wrong with the program or the theory.
>>>
>>> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>>>
>>> I tried two files:
>>>
>>> 1. 212 MB AVI (video only)
>>>
>>> Entropy: 6.211472 bits per byte.
>>>
>>> Reduction would be 22 percent.
>>>
>>> The same file compressed with WinRar:
>>>
>>> 79.2 MB, that's clearly more than 50% compression.
>>>
>>> 2. 900 KB BMP (Picture)
>>>
>>> Reduction would be something like 15 procent.
>>>
>>> It was way over 50 procent
>>>
>>> WinRar file only 300 KB or so.
>>>
>>> Try it yourself I would say
>>>
>>> Bye,
>>> Skybuck.
>>>
>>> "William" <william_156 RemoveThis @gmail.com> wrote in message
>>> news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
>>>> No. Your dead wrong there. Have you even tried this?
>>>>
>>>> Optimum compression means the theoretical maximum you could (losslessy)
>>>> compress a file.
>>>>
>>>> No program could compress file a file filled with random bytes. If it
>>>> could then by definition the data would not be random.
>>>>
>>>> That's how compression works - by finding patterns!
>>>>
>>>> Skybuck Flying wrote:
>>>>> What is ment with optimum compression ?
>>>>>
>>>>> Whatever the case may be, the program is useless to determine what the
>>>>> maximum compressed file size would be.
>>>>>
>>>>> WinZip and WinRar for example compress much better than ent.exe
>>>>> indicates should be theoretically possible.
>>>>>
>>>>> Bye,
>>>>> Skybuck.
>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

In article <tuWdncNZC7QzjObaRVn_vwA RemoveThis @giganews.com>, william_156 RemoveThis @gmail.com
says...
> I stand corrected. This goes against what I thought I knew about
> compression!
>
> Wikipedia has an article on the subject. But the maths is way over my head.
>
> http://en.wikipedia.org/wiki/Information_entropy
>
> Skybuck Flying wrote:
> > I have my own webdrive, I put a BMP on it for you, and the compressed RAR
> > file and the TEXTENT.TXT which contains the output of the "ent.exe TEST.BMP"
> >
> > http://members.home.nl/hbthouppermans/EntropyTest/

The math is over mine as welll, but understanding what's going on in a
real-world implementation of a compression algorithm doesn't require
much math. Just look at how modern compressors work at the bit and byte
level. All they do is look for repeated patterns of bits and bytes. To
take a simple case, if you fill a file with (truly) random 32-bit
unsigned numbers, a fairly large proportion of them will have the top
byte be 0, which is a string of 8 repeated 0 bits. That common value
will be represented by a shorter symbol. The same type of thing applies
to other patterns it may find in the raw bits and bytes, including
patterns which run from one value in the file to the next.

Many long repeating patterns give high compression ratios, while fewer
and shorter patterns give lower compression ratios, but there are always
going to be compressible blocks in any list of random numbers.


> >
> > Test.bmp is 900 KB
> > Test.rar is 347 KB
> >
> > "Ent.exe Test.bmp" output:
> >
> > Entropy = 6.723972 bits per byte.
> >
> > Optimum compression would reduce the size
> > of this 921654 byte file by 15 percent.
> >
> > Chi square distribution for 921654 samples is 2744487.23, and randomly
> > would exceed this value 0.01 percent of the times.
> >
> > Arithmetic mean value of data bytes is 40.3445 (127.5 = random).
> > Monte Carlo value for Pi is 3.819698065 (error 21.58 percent).
> > Serial correlation coefficient is 0.843178 (totally uncorrelated = 0.0).
> >
> > 15 procent of 900 KB = 135 KB
> >
> > Clearly WinRar compressess better than that.
> >
> > Bye,
> > Skybuck.
> >
> > "William" <william_156 RemoveThis @gmail.com> wrote in message
> > news:EpSdnSS4O9HFbefa4p2dnAA@giganews.com...
> >> Simply put I don't believe you. Can you please upload a sample avi and
> >> compressed archive to a service like rapidshare.
> >>
> >> Here is a 10k file filled with random noise from delphis random number
> >> generator:
> >>
> >> http://rapidshare.de/files/38183629/rand.bin.html
> >>
> >> Try and compress it , can't be done!
> >>
> >> Files with high entropy can't be compressed for the same reason you when

--
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Skybuck Flying wrote:
> Well something is wrong with the program or the theory.
>
> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>
> I tried two files:
>
> 1. 212 MB AVI (video only)
>
> Entropy: 6.211472 bits per byte.
>
> Reduction would be 22 percent.
>
> The same file compressed with WinRar:
>
> 79.2 MB, that's clearly more than 50% compression.
>
> 2. 900 KB BMP (Picture)
>
> Reduction would be something like 15 procent.
>
> It was way over 50 procent
>
> WinRar file only 300 KB or so.
>
> Try it yourself I would say
>
> Bye,
> Skybuck.
>
> "William" <william_156.TakeThisOut@gmail.com> wrote in message
> news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
>> No. Your dead wrong there. Have you even tried this?
>>
>> Optimum compression means the theoretical maximum you could (losslessy)
>> compress a file.
>>
>> No program could compress file a file filled with random bytes. If it
>> could then by definition the data would not be random.
>>
>> That's how compression works - by finding patterns!
>>
>> Skybuck Flying wrote:
>>> What is ment with optimum compression ?
>>>
>>> Whatever the case may be, the program is useless to determine what the
>>> maximum compressed file size would be.
>>>
>>> WinZip and WinRar for example compress much better than ent.exe indicates
>>> should be theoretically possible.

You have this all backwards.

It is not an issue if this program or that program produces better
compression. The ent program is a test for "randomness like"
properties. If the ent program can perform some significant amount of
compression (like 1 or 2 percent for small files or even less for
larger files) then that file's data is significantly non-random.

That is all the test does.

The file IS non-random. The estimates are just that, estimates from a
simple compression program. Of course sophisticated programs will do
better or no body would bother with them.

Don't be deliberately thick.


>>>
>>> Bye,
>>> Skybuck.
>
>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

William wrote:
> I stand corrected. This goes against what I thought I knew about
> compression!
>

Skybuck is going off on a tangent.

The subject is if a file contains random like data or not. The ent
program correctly identifies the avi and bmp as non random. That is all
it has to do. A small amount of compression proves the file is non
random - a simple compression algorithm can do this. It does not need a
sophisticated compression algorithm. That more sophisticated programs
can compress further is irrelevant - all that they prove is that yes ent
got it right - the file is non-random.



> Wikipedia has an article on the subject. But the maths is way over my head.
>
> http://en.wikipedia.org/wiki/Information_entropy
>
> Skybuck Flying wrote:
>> I have my own webdrive, I put a BMP on it for you, and the compressed
>> RAR file and the TEXTENT.TXT which contains the output of the "ent.exe
>> TEST.BMP"
>>
>> http://members.home.nl/hbthouppermans/EntropyTest/
>>
>> Test.bmp is 900 KB
>> Test.rar is 347 KB
>>
>> "Ent.exe Test.bmp" output:
>>
>> Entropy = 6.723972 bits per byte.
>>
>> Optimum compression would reduce the size
>> of this 921654 byte file by 15 percent.
>>
>> Chi square distribution for 921654 samples is 2744487.23, and randomly
>> would exceed this value 0.01 percent of the times.
>>
>> Arithmetic mean value of data bytes is 40.3445 (127.5 = random).
>> Monte Carlo value for Pi is 3.819698065 (error 21.58 percent).
>> Serial correlation coefficient is 0.843178 (totally uncorrelated = 0.0).
>>
>> 15 procent of 900 KB = 135 KB
>>
>> Clearly WinRar compressess better than that.
>>
>> Bye,
>> Skybuck.
>>
>> "William" <william_156 RemoveThis @gmail.com> wrote in message
>> news:EpSdnSS4O9HFbefa4p2dnAA@giganews.com...
>>> Simply put I don't believe you. Can you please upload a sample avi
>>> and compressed archive to a service like rapidshare.
>>>
>>> Here is a 10k file filled with random noise from delphis random
>>> number generator:
>>>
>>> http://rapidshare.de/files/38183629/rand.bin.html
>>>
>>> Try and compress it , can't be done!
>>>
>>> Files with high entropy can't be compressed for the same reason you
>>> when you zip a zip file it won't get smaller. Think about it.
>>>
>>>
>>>
>>> Skybuck Flying wrote:
>>>> Well something is wrong with the program or the theory.
>>>>
>>>> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>>>>
>>>> I tried two files:
>>>>
>>>> 1. 212 MB AVI (video only)
>>>>
>>>> Entropy: 6.211472 bits per byte.
>>>>
>>>> Reduction would be 22 percent.
>>>>
>>>> The same file compressed with WinRar:
>>>>
>>>> 79.2 MB, that's clearly more than 50% compression.
>>>>
>>>> 2. 900 KB BMP (Picture)
>>>>
>>>> Reduction would be something like 15 procent.
>>>>
>>>> It was way over 50 procent
>>>>
>>>> WinRar file only 300 KB or so.
>>>>
>>>> Try it yourself I would say
>>>>
>>>> Bye,
>>>> Skybuck.
>>>>
>>>> "William" <william_156 RemoveThis @gmail.com> wrote in message
>>>> news:sq6dnX0OOfVOeufaRVn_vwA@giganews.com...
>>>>> No. Your dead wrong there. Have you even tried this?
>>>>>
>>>>> Optimum compression means the theoretical maximum you could
>>>>> (losslessy) compress a file.
>>>>>
>>>>> No program could compress file a file filled with random bytes. If
>>>>> it could then by definition the data would not be random.
>>>>>
>>>>> That's how compression works - by finding patterns!
>>>>>
>>>>> Skybuck Flying wrote:
>>>>>> What is ment with optimum compression ?
>>>>>>
>>>>>> Whatever the case may be, the program is useless to determine what
>>>>>> the maximum compressed file size would be.
>>>>>>
>>>>>> WinZip and WinRar for example compress much better than ent.exe
>>>>>> indicates should be theoretically possible.
>>>>>>
>>>>>> Bye,
>>>>>> Skybuck.
>>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

The ent program also mentions:

"Optimum compression"

What's that suppose to mean huh ? =D

Either it's optimal or not, could have fooled me =D

Bye,
Skybuck.
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Am Wed, 2 Jan 2008 02:41:44 +0100 schrieb Skybuck Flying:

> Well something is wrong with the program or the theory.
>
> I run ent.exe in a dos box on Windows XP x64 Pro Edition.
>
> I tried two files:
>
> 1. 212 MB AVI (video only)
>
> Entropy: 6.211472 bits per byte.
>
> Reduction would be 22 percent.
>
> The same file compressed with WinRar:
>
> 79.2 MB, that's clearly more than 50% compression.
>
> 2. 900 KB BMP (Picture)
>
> Reduction would be something like 15 procent.
>
> It was way over 50 procent
>
> WinRar file only 300 KB or so.
>
> Try it yourself I would say

WinRAR has some "special" function for compressing mediafiles. If
mediacompression is activated, then you have there a compression with a
loss of detail. Just like converting your BMP to a JPEG or similar, but not
that strong with artifacts. Same could be for the AVI. Make a bit
comparisson of the original file and the packed-unpacked exemplar of it.

Bye
Norbert
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Nonsense at least for WinRar 3.51

Bye,
Skybuck.
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Am Sun, 6 Jan 2008 15:32:02 +0100 schrieb Skybuck Flying:

> Nonsense at least for WinRar 3.51

There was once... But it is long time ago I used winRAR... It is long time
ago I used Windows it self I am forced to it now and here, in any other
case I had to live without internet access...
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

Norbert Melzer wrote:
> Am Sun, 6 Jan 2008 15:32:02 +0100 schrieb Skybuck Flying:
>
>> Nonsense at least for WinRar 3.51
>
> There was once... But it is long time ago I used winRAR... It is long time
> ago I used Windows it self I am forced to it now and here, in any other
> case I had to live without internet access...


all of the random number test programs need a large sample.

just a few 100 numbers out of 2 billion number prove nothing.

there is a relationship between the number of samples and the number of
possible numbers that can be generated.

Jim P.
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix.. 

On 2 Gen, 03:00, William <william_... RemoveThis @gmail.com> wrote:
> Simply put I don't believe you. Can you please upload a sample avi and
> compressed archive to a service like rapidshare.
>
> Here is a 10k file filled with random noise from delphis random number
> generator:
>
> http://rapidshare.de/files/38183629/rand.bin.html
>
> Try and compress it , can't be done!
>
> Files with high entropy can't be compressed for the same reason you when
> you zip a zip file it won't get smaller. Think about it.

Files (that is, sequences of bits) don't have entropy.
Processes generating sequences of bits do.

The Ent program tries to estimate the entropy of the source from the
statistical properties of a sequence it produced.
I suspect it does that under the assumption that every byte (non-
overlapping 8-bit block) of the sequence is uncorrelated. In doing so,
it misses any possible long-distance correlation between bits and thus
overestimates the entropy of the source, hence underestimating the
maximum compressability.

Rar searches for pattens in bit strings way much longer than 8 bits,
and thus is much better at finding long-distance patterns that can be
exploited for compression (those patterns result from long-distance
correlations of the source).
Since typical bitmap images, both computer-generated or photos,
contain lots of long-distance patterns, Rar compresses them much
better than Ent estimates.

<snip>
 >> Stay informed about: Skybuck's Racing Random Number Generator V3 (algorithm fix..