JSON que difiere textualmente

Como parte de mis procesos de lanzamiento, tengo que comparar algunos datos de configuración JSON utilizados por mi aplicación. Como primer bash, simplemente imprimí el JSON y lo difuminé (usando kdiff3 o solo diff).

Sin embargo, a medida que los datos crecieron, kdiff3 confunde diferentes partes en la salida, lo que hace que las adiciones parezcan modificaciones gigantes, eliminaciones impares, etc. Hace que sea muy difícil descubrir qué es diferente. También he probado otras herramientas de diferencias (meld, kompare, diff, algunas otras), pero todas tienen el mismo problema.

A pesar de mis mejores esfuerzos, parece que no puedo formatear el JSON de una manera que las herramientas de diferencias puedan entender.

Datos de ejemplo:

[ { "name": "date", "type": "date", "nullable": true, "state": "enabled" }, { "name": "owner", "type": "string", "nullable": false, "state": "enabled", } ...lots more... ] 

Lo anterior probablemente no causaría el problema (el problema ocurre cuando comienzan a haber cientos de líneas), pero eso es lo esencial de lo que se está comparando.

Eso es sólo una muestra; los objetos completos son atributos 4-5, y algunos atributos tienen atributos 4-5 en ellos. Los nombres de los atributos son bastante uniformes, pero sus valores son muy variados.

En general, parece que todas las herramientas de diferencias confunden el cierre “}” con el cierre de los siguientes objetos “}”. Parece que no puedo romperlos de este hábito.

He intentado agregar espacios en blanco, cambiar la sangría y agregar algunas cadenas “BEGIN” y “END” antes y después de los objetos respectivos, pero la herramienta aún se confunde.

Si cualquiera de sus herramientas tiene la opción, Patience Diff podría funcionar mucho mejor para usted. Trataré de encontrar una herramienta con él (otro tha Git y Bazaar) e informaré.

Edit: Parece que la implementación en Bazaar se puede usar como una herramienta independiente con cambios mínimos.

Edit2: WTH, ¿por qué no pegas la fuente del nuevo script de diff cool que me hiciste hackear? Aquí está, no hay reclamo de derechos de autor por mi parte, es solo el código de Bram / Canonical reorganizado.

 #!/usr/bin/env python # Copyright (C) 2005, 2006, 2007 Canonical Ltd # Copyright (C) 2005 Bram Cohen, Copyright (C) 2005, 2006 Canonical Ltd # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA import os import sys import time import difflib from bisect import bisect __all__ = ['PatienceSequenceMatcher', 'unified_diff', 'unified_diff_files'] py3k = False try: xrange except NameError: py3k = True xrange = range # This is a version of unified_diff which only adds a factory parameter # so that you can override the default SequenceMatcher # this has been submitted as a patch to python def unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n', sequencematcher=None): r""" Compare two sequences of lines; generate the delta as a unified diff. Unified diffs are a compact way of showing line changes and a few lines of context. The number of context lines is set by 'n' which defaults to three. By default, the diff control lines (those with ---, +++, or @@) are created with a trailing newline. This is helpful so that inputs created from file.readlines() result in diffs that are suitable for file.writelines() since both the inputs and outputs have trailing newlines. For inputs that do not have trailing newlines, set the lineterm argument to "" so that the output will be uniformly newline free. The unidiff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for 'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'. The modification times are normally expressed in the format returned by time.ctime(). Example: >>> for line in unified_diff('one two three four'.split(), ... 'zero one tree four'.split(), 'Original', 'Current', ... 'Sat Jan 26 23:30:50 1991', 'Fri Jun 06 10:20:52 2003', ... lineterm=''): ... print line --- Original Sat Jan 26 23:30:50 1991 +++ Current Fri Jun 06 10:20:52 2003 @@ -1,4 +1,4 @@ +zero one -two -three +tree four """ if sequencematcher is None: import difflib sequencematcher = difflib.SequenceMatcher if fromfiledate: fromfiledate = '\t' + str(fromfiledate) if tofiledate: tofiledate = '\t' + str(tofiledate) started = False for group in sequencematcher(None,a,b).get_grouped_opcodes(n): if not started: yield '--- %s%s%s' % (fromfile, fromfiledate, lineterm) yield '+++ %s%s%s' % (tofile, tofiledate, lineterm) started = True i1, i2, j1, j2 = group[0][3], group[-1][4], group[0][5], group[-1][6] yield "@@ -%d,%d +%d,%d @@%s" % (i1+1, i2-i1, j1+1, j2-j1, lineterm) for tag, i1, i2, j1, j2 in group: if tag == 'equal': for line in a[i1:i2]: yield ' ' + line continue if tag == 'replace' or tag == 'delete': for line in a[i1:i2]: yield '-' + line if tag == 'replace' or tag == 'insert': for line in b[j1:j2]: yield '+' + line def unified_diff_files(a, b, sequencematcher=None): """Generate the diff for two files. """ mode = 'rb' if py3k: mode = 'r' # Should this actually be an error? if a == b: return [] if a == '-': file_a = sys.stdin time_a = time.time() else: file_a = open(a, mode) time_a = os.stat(a).st_mtime if b == '-': file_b = sys.stdin time_b = time.time() else: file_b = open(b, mode) time_b = os.stat(b).st_mtime # TODO: Include fromfiledate and tofiledate return unified_diff(file_a.readlines(), file_b.readlines(), fromfile=a, tofile=b, sequencematcher=sequencematcher) def unique_lcs_py(a, b): """Find the longest common subset for unique lines. :param a: An indexable object (such as string or list of strings) :param b: Another indexable object (such as string or list of strings) :return: A list of tuples, one for each line which is matched. [(line_in_a, line_in_b), ...] This only matches lines which are unique on both sides. This helps prevent common lines from over influencing match results. The longest common subset uses the Patience Sorting algorithm: http://en.wikipedia.org/wiki/Patience_sorting """ # set index[line in a] = position of line in a unless # a is a duplicate, in which case it's set to None index = {} for i in xrange(len(a)): line = a[i] if line in index: index[line] = None else: index[line]= i # make btoa[i] = position of line i in a, unless # that line doesn't occur exactly once in both, # in which case it's set to None btoa = [None] * len(b) index2 = {} for pos, line in enumerate(b): next = index.get(line) if next is not None: if line in index2: # unset the previous mapping, which we now know to # be invalid because the line isn't unique btoa[index2[line]] = None del index[line] else: index2[line] = pos btoa[pos] = next # this is the Patience sorting algorithm # see http://en.wikipedia.org/wiki/Patience_sorting backpointers = [None] * len(b) stacks = [] lasts = [] k = 0 for bpos, apos in enumerate(btoa): if apos is None: continue # as an optimization, check if the next line comes at the end, # because it usually does if stacks and stacks[-1] < apos: k = len(stacks) # as an optimization, check if the next line comes right after # the previous line, because usually it does elif stacks and stacks[k] < apos and (k == len(stacks) - 1 or stacks[k+1] > apos): k += 1 else: k = bisect(stacks, apos) if k > 0: backpointers[bpos] = lasts[k-1] if k < len(stacks): stacks[k] = apos lasts[k] = bpos else: stacks.append(apos) lasts.append(bpos) if len(lasts) == 0: return [] result = [] k = lasts[-1] while k is not None: result.append((btoa[k], k)) k = backpointers[k] result.reverse() return result def recurse_matches_py(a, b, alo, blo, ahi, bhi, answer, maxrecursion): """Find all of the matching text in the lines of a and b. :param a: A sequence :param b: Another sequence :param alo: The start location of a to check, typically 0 :param ahi: The start location of b to check, typically 0 :param ahi: The maximum length of a to check, typically len(a) :param bhi: The maximum length of b to check, typically len(b) :param answer: The return array. Will be filled with tuples indicating [(line_in_a, line_in_b)] :param maxrecursion: The maximum depth to recurse. Must be a positive integer. :return: None, the return value is in the parameter answer, which should be a list """ if maxrecursion < 0: print('max recursion depth reached') # this will never happen normally, this check is to prevent DOS attacks return oldlength = len(answer) if alo == ahi or blo == bhi: return last_a_pos = alo-1 last_b_pos = blo-1 for apos, bpos in unique_lcs_py(a[alo:ahi], b[blo:bhi]): # recurse between lines which are unique in each file and match apos += alo bpos += blo # Most of the time, you will have a sequence of similar entries if last_a_pos+1 != apos or last_b_pos+1 != bpos: recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1, apos, bpos, answer, maxrecursion - 1) last_a_pos = apos last_b_pos = bpos answer.append((apos, bpos)) if len(answer) > oldlength: # find matches between the last match and the end recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1, ahi, bhi, answer, maxrecursion - 1) elif a[alo] == b[blo]: # find matching lines at the very beginning while alo < ahi and blo < bhi and a[alo] == b[blo]: answer.append((alo, blo)) alo += 1 blo += 1 recurse_matches_py(a, b, alo, blo, ahi, bhi, answer, maxrecursion - 1) elif a[ahi - 1] == b[bhi - 1]: # find matching lines at the very end nahi = ahi - 1 nbhi = bhi - 1 while nahi > alo and nbhi > blo and a[nahi - 1] == b[nbhi - 1]: nahi -= 1 nbhi -= 1 recurse_matches_py(a, b, last_a_pos+1, last_b_pos+1, nahi, nbhi, answer, maxrecursion - 1) for i in xrange(ahi - nahi): answer.append((nahi + i, nbhi + i)) def _collapse_sequences(matches): """Find sequences of lines. Given a sequence of [(line_in_a, line_in_b),] find regions where they both increment at the same time """ answer = [] start_a = start_b = None length = 0 for i_a, i_b in matches: if (start_a is not None and (i_a == start_a + length) and (i_b == start_b + length)): length += 1 else: if start_a is not None: answer.append((start_a, start_b, length)) start_a = i_a start_b = i_b length = 1 if length != 0: answer.append((start_a, start_b, length)) return answer def _check_consistency(answer): # For consistency sake, make sure all matches are only increasing next_a = -1 next_b = -1 for (a, b, match_len) in answer: if a < next_a: raise ValueError('Non increasing matches for a') if b < next_b: raise ValueError('Non increasing matches for b') next_a = a + match_len next_b = b + match_len class PatienceSequenceMatcher_py(difflib.SequenceMatcher): """Compare a pair of sequences using longest common subset.""" _do_check_consistency = True def __init__(self, isjunk=None, a='', b=''): if isjunk is not None: raise NotImplementedError('Currently we do not support' ' isjunk for sequence matching') difflib.SequenceMatcher.__init__(self, isjunk, a, b) def get_matching_blocks(self): """Return list of triples describing matching subsequences. Each triple is of the form (i, j, n), and means that a[i:i+n] == b[j:j+n]. The triples are monotonically increasing in i and in j. The last triple is a dummy, (len(a), len(b), 0), and is the only triple with n==0. >>> s = PatienceSequenceMatcher(None, "abxcd", "abcd") >>> s.get_matching_blocks() [(0, 0, 2), (3, 2, 2), (5, 4, 0)] """ # jam 20060525 This is the python 2.4.1 difflib get_matching_blocks # implementation which uses __helper. 2.4.3 got rid of helper for # doing it inline with a queue. # We should consider doing the same for recurse_matches if self.matching_blocks is not None: return self.matching_blocks matches = [] recurse_matches_py(self.a, self.b, 0, 0, len(self.a), len(self.b), matches, 10) # Matches now has individual line pairs of # line A matches line B, at the given offsets self.matching_blocks = _collapse_sequences(matches) self.matching_blocks.append( (len(self.a), len(self.b), 0) ) if PatienceSequenceMatcher_py._do_check_consistency: if __debug__: _check_consistency(self.matching_blocks) return self.matching_blocks unique_lcs = unique_lcs_py recurse_matches = recurse_matches_py PatienceSequenceMatcher = PatienceSequenceMatcher_py def main(args): import optparse p = optparse.OptionParser(usage='%prog [options] file_a file_b' '\nFiles can be "-" to read from stdin') p.add_option('--patience', dest='matcher', action='store_const', const='patience', default='patience', help='Use the patience difference algorithm') p.add_option('--difflib', dest='matcher', action='store_const', const='difflib', default='patience', help='Use python\'s difflib algorithm') algorithms = {'patience':PatienceSequenceMatcher, 'difflib':difflib.SequenceMatcher} (opts, args) = p.parse_args(args) matcher = algorithms[opts.matcher] if len(args) != 2: print('You must supply 2 filenames to diff') return -1 for line in unified_diff_files(args[0], args[1], sequencematcher=matcher): sys.stdout.write(line) if __name__ == '__main__': sys.exit(main(sys.argv[1:])) 

Edición 3: También he realizado una versión mínimamente independiente de Diff Match and Patch de Neil Fraser , me gustaría mucho una comparación de resultados para su caso de uso. Una vez más, no reclamo derechos de autor.

Edición 4: acabo de encontrar DataDiff , que podría ser otra herramienta para probar.

DataDiff es una biblioteca para proporcionar diffs legibles por humanos de estructuras de datos de python. Puede manejar tipos de secuencia (listas, tuplas, etc.), conjuntos y diccionarios.

Los diccionarios y las secuencias se difundirán recursivamente, cuando corresponda.

Entonces, escribí una herramienta para hacer diferencias unificadas de archivos JSON hace un tiempo que podría ser de algún interés.

https://github.com/jclulow/jsondiff

Algunos ejemplos de entrada y salida para la herramienta aparecen en la página de github.

Usted debe retirar el difusor de subestaca. Es tanto un módulo node.js como una utilidad de línea de comandos que hace exactamente esto:

https://github.com/substack/difflet

Sé que esta es una pregunta bastante antigua, pero el módulo de Python “Herramientas JSON” ofrece otra solución para diferenciar archivos json:

https://pypi.python.org/pypi/json_tools https://bitbucket.org/vadim_semenov/json_tools/src/75cc15381188c760badbd5b66aef9941a42c93fa?at=default

Eclipse podría hacerlo mejor. Abra los dos archivos en un proyecto de eclipse, selecciónelos y haga clic con el botón derecho -> comparar -> entre sí.

Más allá de los cambios de formato, la herramienta de diferenciación también debe ordenar las propiedades de los objetos JSON de manera estable (alfabéticamente, por ejemplo), ya que el orden de las propiedades es semánticamente sin sentido. Es decir, la reordenación de las propiedades no debe cambiar el significado de los contenidos.

Aparte de esto, el análisis y la impresión bonita de una manera que coloca a lo sumo una entrada en una sola línea podría permitir el uso de diferencias textuales. De lo contrario, cualquier algoritmo diff que funcione en árboles (que se usa para la diferencia de xml) debería funcionar mejor.