#!/usr/bin/env perl

=pod

=head1 NAME

te (table editor) - front-end that simplifies editing tab-delimited

text tables

=head1 SYNOPSIS

B<te> I<file1> [ I<file2> ... ]

Run without arguments to see options.

=head1 DESCRIPTION

This program makes it easier to edit tab-delimited text tables, such

as are used with Interchange (see interchangecommerce.org), and can be

exported from many popular spreadsheet and database applications.

It converts tab-delimited text files that have one record per line into

temporary files with one field per line, each line beginning with the

field name. It then sends each file to your selected text editor. After

you exit your editor, it checks to see if you changed anything in the

file, and if so, it converts the data back to the tab-delimited format

with one record per line, and replaces the original file.

The first line of each input file must contain the field names,

tab-delimited, that apply for that file.

Editing is pretty straightforward when you see it in action. The

rules are:

=over 4

=item o

Empty lines are ignored.

=item o

Comment lines (beginning with C<#>) are ignored at the beginning of the

file, and terminate a record in the middle.

=item o

The fields in the first record are used in all subsequent records,

and the order in which they are specified are the order in which the

columns will be written.

=item o

To delete a column, delete its line in the first record.

=item o

To change which order the columns are in, re-order the first record.

=item o

To add a column, add it to the first record (and as many subsequent

records as you wish).

Note that if you're using the "extended" option (-e), you can't add new

columns, because they can't be distinguished from the extended fields that

go into the serialized hash. You would need to do that in a separate pass.

=item o

If you delete a column, you do not need to delete it from every record;

any instances of that field in records after the first will be ignored.

=item o

Any space left after the field name and colon (like C<fieldname:>) will be

included as part of the field. Any tabs you put in the field data itself

will be converted to spaces (since they would corrupt the table otherwise).

=item o

A record can be deleted by removing all its fields. A new record can be

added by inserting a new block of all fields at a record boundary.

=item o

If any errors are encountered, such as non-existent field names or lines

that don't follow the prescribed format, processing aborts immediately

and the original file is left untouched.

=back

You can edit several files in succession by naming each on the command

line. The editor will be called for each one independently. If you start

editing many files and decide you want to stop, add a line C<#DONE>

anywhere in the temporary file and save it. The current file will be

processed and saved, but the rest will be skipped.

As is customary with many Unix applications, you can set the environment

variables VISUAL or EDITOR to point to your favorite text editor. If

neither of those is set, B<vi>(1) is used.

Options will also be read from environment variable TE_OPTIONS if it is

set.

=head1 AUTHOR

Jon Jensen <jonj@cpan.org>

=head1 COPYRIGHT

Copyright (C) 2002-2017 Jon Jensen and others

Copyright (C) 2001-2002 Red Hat, Inc.

This program is free software; you can redistribute it and/or modify it

under the terms of the GNU General Public License as published by the

Free Software Foundation; either version 2 of the License, or (at your

option) any later version.

This program is distributed in the hope that it will be useful, but

WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY

or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License

at http://www.fsf.org/copyleft/gpl.html for more details.

=head1 VERSION

2017-08-09

=head1 CHANGELOG

2001-04-26. Initial release.

2001-05-04. Make temporary file in working directory instead of using

POSIX::tmpnam. Keep ownership and permissions of original file. Fixed

bug that caused fields with number "0" to be output as empty strings.

2001-05-06. Fix problem with how temporary file names were generated.

2001-07-31. Check for VISUAL environment variable, and handle editor

options there or in EDITOR if given. Take advantage of Digest::MD5's

native file reading instead of doing it ourselves.

2001-10-03. Added gvim support by forcing foreground option -f.

2002-07-23. Remove line endings whether CR, LF, or CRLF, instead of

using running Perl's platform-specific chomp. If preserving a trailing

solitary CR in the last field of a line is important, you'll want to

change this behavior.

2002-08-30. Add option -s for starting value support (really only vi).

te -s os28004 <file>

Jumps to first occurrence of "os28004" in <file>. Option -i ignores case

in the search. (By Mike Heins.)

2002-09-02. Add option -f to handle files without field names.

2002-09-03. Add option -n to number rows in comments. Allow setting of

persistent options in environment variable TE_OPTIONS.

2004-06-07. Fixed bug that misinterpreted file as having no data rows

when last line of file was empty.

2005-08-29. Added ability to delete, re-order, or add columns by placing

them in the first record.

2005-11-15. Added support for extended columns containing Perl

serialized hashes with the -e option.

2008-04-11. Added option -o to write output to a file and exit, never

invoking an editor. By Greg Sabino Mullane.

2017-08-09. Added option -t to suppress tabs for empty final columns.

Added option -h to suppress output of an empty extended attribute hash {}.

Started sorting extended attributes in stringified hashes to have a

stable order, since newer Perl hash key ordering is random per run.

Made agnostic about character set encoding.

Use Safe reval instead of unsafe eval on extended attribute serialized hashes

to avoid running possibly untrusted code.

Updated code to modernize some Perl conventions: 3-arg open, lexical

filehandles, etc.

=cut

use strict;

use warnings;

use 5.010_000;

use Digest::MD5;

use File::Basename 'fileparse';

use Text::ParseWords;

use Data::Dumper;

use Safe;

use Getopt::Std;

my ($prog) = fileparse($0);

my $USAGE = <<EOF;

Usage: $prog [-i] [-s startpoint] tablefile1 [tablefile2 ...]

Edit tab-delimited file with easy field name delineation.

Options:

-i Ignores case on vim jump search.

-s TEXT Jumps to first line where TEXT is. Only for vi.

-f Do not look for field names on first line of file.

-n Number rows in comments

-e field Extra fields in this field, a stringified hash

-h Suppress empty extended attributes hash (with -e)

-t Trim end-of-line tabs around empty fields

-o FILE Convert to the named output file and exit

See 'man te' or 'perldoc $0' for more information.

EOF

unshift @ARGV, Text::ParseWords::shellwords($ENV{TE_OPTIONS})

if defined $ENV{TE_OPTIONS};

use vars qw/$opt_i $opt_s $opt_f $opt_n $opt_e $opt_h $opt_o $opt_t/;

getopts('is:fne:ho:t') or die "$@\n$USAGE";

die $USAGE unless @ARGV;

my @ed = Text::ParseWords::shellwords($ENV{VISUAL} || $ENV{EDITOR} || 'vi');

if ($opt_s) {

if ($opt_i) {

push @ed, '-c', 'set ic';

$opt_s = lc $opt_s;

}

$opt_s =~ s:/:\\/:g;

push @ed, '-c', qq{/$opt_s/};

}

# run gvim in foreground mode, since it otherwise immediately returns

# control to us and we never get the user's changes

if($ed[0] =~ /\bgvim\b/) {

push @ed, '-f' unless grep $_ eq '-f', @ed;

}

my $compartment;

if ($opt_e) {

$compartment = Safe->new;

}

for my $filename (@ARGV) {

my (@fieldnames, $fieldcount, @fields);

my ($name, $path, $in, $out, $tmpfile, $newfile, $digest1, $digest2);

unless (-e $filename) {

warn "Skipping '$filename': file does not exist\n";

next;

}

unless (-f $filename) {

warn "Skipping '$filename': not a regular file\n";

next;

}

unless (open $in, '<', $filename) {

warn "Error opening '$filename' for reading: $!\n";

next;

}

binmode $in;

# get field names

$_ = <$in>;

s/\x0d?\x0a?$//;

$fieldcount = tr/\t/\t/ + 1;

if ($opt_f) {

@fieldnames = map { "field$_" } (1 .. $fieldcount);

seek $in, 0, 0;

}

else {

die "Error in '$filename' header: null field name found\n" if /\t\t/;

@fieldnames = split /\t/, $_, $fieldcount;

}

my %fieldnames = map { $_ => 1 } @fieldnames;

($name, $path) = fileparse($filename);

# I tried keeping the whole file in memory (for MD5's sake) instead of

# first writing to disk, but doing it this way turned out to be about 5

# times faster and used 1/10th the memory on large files. (My benchmark

# was a 12 MB products.txt table for Interchange.)

$tmpfile = "$path.$name.tmp.$$";

if ($opt_o) {

if (-e $opt_o) {

print "Overwrite $opt_o? ";

my $ans = <STDIN>;

if ($ans !~ /^Y/i) {

print "\nFile not overwritten, exiting.\n";

exit;

}

}

$tmpfile = $opt_o;

}

open $out, '>', $tmpfile or die "Error opening '$tmpfile' for writing: $!\n";

binmode $out;

print STDERR "Prettifying $filename\n";

print $out <<EOF;

#

# This is a temporary file, automatically generated from the data file:

#

# $filename

#

# If you change anything in it, it will be converted back into the original

# format and will replace the original file.

#

EOF

my $rowcount = 0;

while (<$in>) {

s/\x0d?\x0a?$//;

++$rowcount, print $out "# row $rowcount\n" if $opt_n;

@fields = split /\t/, $_, $fieldcount;

my $extended;

for (my $i = 0; $i < @fieldnames; $i++) {

$extended = $i, next if $opt_e and $fieldnames[$i] eq $opt_e;

print $out $fieldnames[$i], ":", $fields[$i] // '', "\n";

}

if ($opt_e) {

die "Extended field '$opt_e' does not exist\n"

unless $fieldnames{$opt_e};

if ($fields[$extended]) {

my $extra = $compartment->reval($fields[$extended]);

if (ref($extra) eq 'HASH') {

for (sort keys %$extra) {

if ($fieldnames{$_}) {

print $out <<EOF;

# NOTE! The following field '$_' from the serialized hash

# in field '$opt_e' duplicates one of the base columns.

# If duplicates exist when saving, the last one encountered will win.

EOF

}

print $out $_, ":", $extra->{$_}, "\n";

}

}

else {

die "Invalid extended field '$opt_e': $fields[$extended]\n";

}

}

}

print $out "#\n";

}

my $have_rows = ($. > 1);

if ($have_rows) {

print $out <<EOF;

# You can uncomment the following lines to use as a template for inserting

# a new row into the table. Copy as many times as needed to add many rows.

#

EOF

}

else {

print $out <<EOF;

# Your file was empty -- it had no data rows, only field definitions.

# You can copy the following empty row template as many times as needed

# to add new rows to the table.

#

EOF

}

print $out join("\n", map { ($have_rows ? '#' : '') . $_ . ":" } @fieldnames);

print $out "\n#\n";

close $in;

print $out <<EOF;

# end of file

#

EOF

close $out or die "Error closing '$tmpfile' after writing: $!\n";

if ($opt_o) {

print "Wrote $opt_o\n";

exit;

}

open $in, '<', $tmpfile or die "Error opening '$tmpfile' for reading: $!\n";

binmode $in;

$digest1 = Digest::MD5->new->addfile($in)->digest;

close $in;

system (@ed, $tmpfile) == 0

or do {

for (@ed) {

next unless /\s/;

s/"/\\"/g;

$_ = qq["$_"];

}

my $editor = join ' ', @ed;

die "Error calling editor '$editor' with '$tmpfile': $!\n";

};

open $in, '<', $tmpfile or die "Error opening '$tmpfile' for reading: $!\n";

binmode $in;

$digest2 = Digest::MD5->new->addfile($in)->digest;

if ($digest1 eq $digest2) {

print STDERR "No changes made; '$filename' untouched\n";

close $in;

unlink $tmpfile;

next;

}

print STDERR "Importing changes back into '$filename'\n";

$newfile = "$path.$name.new.$$";

open $out, '>', $newfile or die "Error opening '$newfile' for writing: $!\n";

binmode $out;

my @newfields;

my %found_fields;

@found_fields{@fieldnames} = @fieldnames;

my $fields_out = join("\t", @fieldnames) . "\n";

my $tabcounter = 0;

my $fieldpos = 0;

my $done;

seek $in, 0, 0 or die "Error rewinding file '$tmpfile': $!\n";

my %record;

no warnings qw/ uninitialized /;

while (<$in>) {

$done = 1 if /^#\s*DONE/;

if(/^#/) {

next unless $fieldpos;

if ($fields_out) {

if ($opt_e) {

# can't change field list when -e in effect

@newfields = @fieldnames;

}

@found_fields{@newfields} = @newfields;

$fields_out = join("\t", @newfields) . "\n";

print $out $fields_out;

undef $fields_out;

}

if ($opt_e) {

my %extra;

for (keys %record) {

next if $fieldnames{$_};

$extra{$_} = delete $record{$_};

}

if (%extra or ! $opt_h) {

my $d = Data::Dumper->new([ \%extra ]);

$d->Indent(0)->Terse(1)->Sortkeys(1);

$record{$opt_e} = $d->Dump;

}

}

my $out_line = join("\t", @record{@newfields});

$out_line =~ s/\t+$// if $opt_t;

print $out $out_line, "\n";

%record = ();

$fieldpos = 0;

next;

}

next unless /\S/;

/^([^:]+):(.*)$/ or

die "Error parsing line $. of '$tmpfile': line format unknown:\n$_";

my $fn = $1;

if ($fields_out) {

push @newfields, $fn;

}

elsif (! $found_fields{$fn} and ! $opt_e) {

die "Error parsing line $. of '$tmpfile': bad field name '$fn'\n";

}

$_ = $2;

$tabcounter += s/\t/ /g;

$record{$fn} = $_;

$fieldpos++;

}

print STDERR "$tabcounter tab character",

$tabcounter == 1 ? ' was' : 's were',

" found in the data! Each tab was replaced with a space.\n"

if $tabcounter;

close $out or die "Error closing '$filename.new' after writing: $!\n";

close $in or die "Error closing '$tmpfile' after reading: $!\n";

my ($mode, $uid, $gid) = (stat($filename))[2,4,5];

chmod $mode, $newfile;

chown $uid, $gid, $newfile if $> == 0;

rename $newfile, $filename or

die "Error renaming '$newfile' to '$filename': $!\n";

unlink $tmpfile;

if ($done) {

print STDERR "Found 'DONE' command; skipping rest of files.\n";

last;

}

}

# vim: set noet: