[john-dev] 4 failing tests for ppc64 big-endian

[Normand]

Hello there,

I experienced 4 test failures while doing build of john package on openSUSE (1)
for ppc64 (64 bits big-endian) or ppc (32 bits big-endian)
As detailed in (2) I am using the latest jumboversion john-1.7.9-jumbo-7 version

Any suggestion on how to investigate such failures ?


===
$grep -E 'Benchmarking.*FAILED' john_OBS_ppc64_failure.log
[ 103s] Benchmarking: MS SQL 2005 SHA-1 [32/64]... FAILED (get_hash[0](0))
[ 103s] Benchmarking: Office 2007/2010 SHA-1/AES [32/64]... FAILED (cmp_all(1))
[ 104s] Benchmarking: WPA-PSK PBKDF2-HMAC-SHA-1 [32/64]... FAILED (valid)
[ 104s] Benchmarking: RAR3 SHA-1 AES (4 characters) [32/64]... FAILED (cmp_all(1))
===
$grep -E 'Benchmarking.*FAILED' john_OBS_ppc_failure.log
[ 93s] Benchmarking: MS SQL 2005 SHA-1 [32/32]... FAILED (get_hash[0](0))
[ 93s] Benchmarking: Office 2007/2010 SHA-1/AES [32/32]... FAILED (cmp_all(1))
[ 94s] Benchmarking: WPA-PSK PBKDF2-HMAC-SHA-1 [32/32]... FAILED (valid)
[ 94s] Benchmarking: RAR3 SHA-1 AES (4 characters) [32/32]... FAILED (cmp_all(1))
===


(1) https://build.opensuse.org/package/live_build_log/home:michel_mno:branches:security/john/openSUSE_Factory_PowerPC/ppc64
(2) https://build.opensuse.org/package/view_file/home:michel_mno:branches:security/john/john.spec?expand=1


PS: Is there a way to retrieve as one file the all john-devel ML to ease grep of previous discussions ?

[Solar Designer]

Hi Michel,

Thank you for reporting these issues to us, and for offering help.

Post by Normand
I experienced 4 test failures while doing build of john package on openSUSE (1)
for ppc64 (64 bits big-endian) or ppc (32 bits big-endian)
As detailed in (2) I am using the latest jumboversion john-1.7.9-jumbo-7 version

1.7.9-jumbo-7 is not the latest.

The latest jumbo release is 1.8.0-jumbo-1, as listed on JtR homepage.
Unfortunately, to compile it on non-x86 you need this tiny patch:

http://www.openwall.com/lists/john-dev/2014/12/18/21

Development has since proceeded thousands of commits ahead, and we're
close to releasing 1.8.0-jumbo-2 (or possibly 1.8.1-jumbo-1, if 1.8.1 is
released sooner). You may obtain the latest code with:

git clone https://github.com/magnumripper/JohnTheRipper jumbo

Post by Normand
Any suggestion on how to investigate such failures ?

In general, it's tricky to investigate endianness bugs. Short of making
a lucky educated guess as to where the problem is (which does happen),
you'd have to introduce debugging output throughout the computation and
see where it starts to differ between little- and big-endian builds.

In this particular case, though, chances are that those specific issues
are already fixed - so please re-test with the latest from GitHub first.
It is also likely that entirely new issues have since been introduced.

Last summer, Lei Zhang worked with us under GSoC 2015 on adding more use
of AltiVec intrinsics to jumbo. You could want to test this, too.
In fact, the configure script should autodetect AltiVec and use it.

Post by Normand
PS: Is there a way to retrieve as one file the all john-devel ML to ease
grep of previous discussions ?

ezmlm-idx allows you to retrieve 100 messages at a time (via e-mail),
but not the entire thing at once.

You may also use Gmane's search feature here:

http://dir.gmane.org/gmane.comp.security.openwall.john.devel

(and advanced search on the next page, after submitting a search query).

You may also use Google web search like this:

site:openwall.com jumbo ppc

Alexander

[Normand]

Post by Solar Designer
Hi Michel,
Thank you for reporting these issues to us, and for offering help.

Post by Normand
I experienced 4 test failures while doing build of john package on openSUSE (1)
for ppc64 (64 bits big-endian) or ppc (32 bits big-endian)
As detailed in (2) I am using the latest jumboversion john-1.7.9-jumbo-7 version

1.7.9-jumbo-7 is not the latest.
The latest jumbo release is 1.8.0-jumbo-1, as listed on JtR homepage.
http://www.openwall.com/lists/john-dev/2014/12/18/21
Development has since proceeded thousands of commits ahead, and we're
close to releasing 1.8.0-jumbo-2 (or possibly 1.8.1-jumbo-1, if 1.8.1 is
git clone https://github.com/magnumripper/JohnTheRipper jumbo

I started from there, and was able to verify that build and make check
completed for PowerPC Little-Endian with current head of bleeding-jumbo branch
BUT, failed for Big-Endian as per:
===
...
gcc -DAC_BUILT -DJOHN_powerpc64 -c -g -O2 -I/usr/local/include -DARCH_LITTLE_ENDIAN=0 -Wall -Wdeclaration-after-statement -fomit-frame-pointer --param allow-store-data-races=0 -Wno-deprecated-declarations -Wunused-but-set-variable -std=gnu89 -Wdate-time -D_POSIX_SOURCE -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -DOPENSSL_LOAD_CONF -fopenmp -pthread -I/usr/local/include -funroll-loops MD5_fmt.c -o
MD5_fmt.o
In file included from MD5_fmt.c:17:0:
simd-intrinsics.h:79:29: error: expected '}' before 'BITS'
#define MD5_ALGORITHM_NAME BITS " " SIMD_TYPE " " MD5_N_STR
===

Post by Solar Designer

Post by Normand
Any suggestion on how to investigate such failures ?

In general, it's tricky to investigate endianness bugs. Short of making
a lucky educated guess as to where the problem is (which does happen),
you'd have to introduce debugging output throughout the computation and
see where it starts to differ between little- and big-endian builds.
In this particular case, though, chances are that those specific issues
are already fixed - so please re-test with the latest from GitHub first.
It is also likely that entirely new issues have since been introduced.
Last summer, Lei Zhang worked with us under GSoC 2015 on adding more use
of AltiVec intrinsics to jumbo. You could want to test this, too.
In fact, the configure script should autodetect AltiVec and use it.

Looking at git last commit, I identified commit 11b0799 as related work
but seems this is not active for PowerPC Big-Endian.
I tried to make some changes in source code around those files
but did not succeeded as my work results in more than 100 testcases FAILED.

Post by Solar Designer

Post by Normand
PS: Is there a way to retrieve as one file the all john-devel ML to ease
grep of previous discussions ?

ezmlm-idx allows you to retrieve 100 messages at a time (via e-mail),
but not the entire thing at once.
http://dir.gmane.org/gmane.comp.security.openwall.john.devel
(and advanced search on the next page, after submitting a search query).
site:openwall.com jumbo ppc
Alexander