Parse space-delimited text file, store line, re-parse

[mhaber]

Hi guys.
At work we have lots of certificates to sign/encrypt/decrypt different kinds of documents/data and lately there's been a few occasions where we almost forgot to renew them before they expired.
I am currently working on a small monitoring tool, that will notify us of certificates expiring soon.
Trouble is, I only have limited C++ experience and it dates quite a few years back.

Anyway, I have text file that lists all the certificates along with their renewal and expiration date. What I want to do is parse this file, discard the first 5 lines, store each line in a data structure (could be a vector or a dynamic array or a struct) then re-parse every item to extract the NAME, RENEWAL START DATE and CERTIFICATE EXPIRY.
The columns are space separated (it doesn't show in the forum) with exactly the same number on each line. Eg:

HSM1:****[2]HumanPwd[4]Signing[6]06 Apr 2011[5]15 Jul 2011[5]cn=***,o=***,o=***

I realize it's a quite a lot of questions, but it's been a few days and I keep banging my head against the wall..

Profile Name     Profile     Certificate  Certificate     Certificate     User DN       
                 Password    Type         Renewal Start   Private Key          
                 Policy                   Date (GMT)      Expiry (GMT)    
------------------------------------------------------------------------------------------------------------------
HSM1: ****  HumanPwd    Signing      06 Apr 2011     15 Jul 2011     cn=***,o=***,o=***  
HSM1: ****  AppliPwd    Signing      29 Aug 2011     07 Dec 2011     cn=***,o=***,o=***  
HSM1: ****  AppliPwd    Signing      29 Aug 2011     07 Dec 2011     cn=***,o=***,o=***  
HSM1: ****  HumanPwd    Encryption   05 Oct 2011     13 Jan 2012     cn=***,o=***,o=***  
HSM1: ****  HumanPwd    Encryption   12 Jan 2012     21 Apr 2012     cn=***,o=***,o=***

[WaltP]

I realize it's a quite a lot of questions, but it's been a few days and I keep banging my head against the wall..

Actually, there were no questions at all. Just a general description of your project.

It helps to give us the details about what you're banging your head about so we know what you need help with. I hope it's not "discard the first 5 lines" since that's the easiest part of the task.

[mhaber]

Yeah you're right..guess I was just frustrated with c++
Anyway, I'm stuck when trying to fill up the vector with the lines from the text file. This is what I got so far:

#include <iostream>
#include <fstream>
#include <strstream>
#include <stdio.h>
#include <stdlib.h>
#include <string>
#include <vector>

using namespace std;

int main()
{
        int i, vSize;
        int numOfLines = 0;
        string line;


        // Needs to be dynamic
        //char **buffer = NULL;

        ifstream certfiles;
        certfiles.open ("/mnt/SNLDrive/certlistout.txt", ios::in);

        // Check file existence
        if ( !certfiles )
        {
                cout << "Error: Cannot open file!" << endl;
                exit(1);
        }

        while(getline(certfiles, line))
        {
                numOfLines++;
        }

                vector<string> lines;

                while (getline(certfiles, line))
                {
                        lines.push_back(line);
                }

                vSize = lines.size();
                printf("vSize : %d \n", vSize);
                printf("numOfLines : %d \n", numOfLines);

                certfiles.close();

        return 0;
}

Why does vSize return 0?

[pseudorandom21]

The easiest way may be to use the C++ STL.

ifstream inFile();//<-- file opened and checked for errors elsewhere
vector<string> dataStructure;
string input;

//Skip the first 5 lines:
for(int i = 0; i < 5; i++)
  getline(inFile,input);
//Store file in a data structure
while(getline(inFile,input))
   dataStructure.push_back(input);
//Extract some tokens
for(vector<string>::size_type i = 0; i < dataStructure.size(); i++)
{
stringstream ss(dataStructure[i]);
while(ss >> input)
   cout << input;//<-- token
}

Depending on how large the file is are you sure you want to store it in memory? Also, the use of the function "exit" in C++ is likely discouraged as it may result in destructors not being called (don't take my word for it, google it).

Not to be a stickler too but, it's usually considered bad practice to mix C and C++ IO.

vSize is zero because you've already read the entire file and the "get" pointer is at the end of the file, you have to set it back to the beginning to read the file again.
:: someFile.seekg(0,ios::beg);

[mhaber]

Thanks. What you suggested seems to work, I just have a doubt: after doing the ss >> input, I see that all white spaces are trimmed and the lines split into several chucks. I tried copying every chunk into a char array, but I can't seem to be figuring out the cast (eg. for line #7):

char arrBuffer[10][64];
stringstream ss (dataStructure[6]);
          while (ss >> input)
                 for (int k=0; k<=9; k++)
                      arrBuffer[i] = (char) input;

I only need three fields from the textfile ("Profile Name", "Renewal Start Date" and "Expiry"). The dates are split (since there's spaces in the middle), but putting them together shouldn't be too difficult

[pseudorandom21]

Thanks. What you suggested seems to work, I just have a doubt: after doing the ss >> input, I see that all white spaces are trimmed and the lines split into several chucks. I tried copying every chunk into a char array, but I can't seem to be figuring out the cast (eg. for line #7):

char arrBuffer[10][64];
stringstream ss (dataStructure[6]);
          while (ss >> input)
                 for (int k=0; k<=9; k++)
                      arrBuffer[i] = (char) input;

I only need three fields from the textfile ("Profile Name", "Renewal Start Date" and "Expiry"). The dates are split (since there's spaces in the middle), but putting them together shouldn't be too difficult

Why do you want to use character arrays at all? This line is completely nonsensical: arrBuffer[i] = (char) input; because "input" is a C++ string class instance.

Here's a small C++ string class demo I wrote a while back:

#include <iostream>
#include <string>
#include <algorithm>
#include <sstream>
using namespace std;

int main()
{
 string cppString = "Hello World.";
 cppString = "No error, no confusing C functions.";
 getline(cin,cppString);//No buffer overflow from too much input.

 //Use much like an array.
 for( std::string::size_type i = 0; i < cppString.size(); i++ )
   cppString[i] = 'x';

 cppString += " -- Simple string concatenation too.";

 //Usable with the C++ STL algorithms.
 random_shuffle(cppString.begin(), cppString.end());

 cout << "Can still get a C string if you REALLY want to also." 
  << endl << cppString.c_str() << endl;

 //Easily convert from string to int/floating point type.
 int someInt = 0;
 cin >> cppString;
 if((stringstream(cppString) >> someInt).fail())
  cout << "Invalid number, woops!";
 else
  cout << someInt << endl << "Just your lucky number." << endl;
}

If it has a small error it's because I copy&pasted it from my blog.

[Duoas]

You should have yourself a little class that represents your data, and overload the extraction operator to handle it.

Since I was really bored today, I wrote a program for you. Change the parts you need to match your assignment.

Notice that there are several classes in use here.

• date_t
  A very simple date-handling class. There are a lot of other ways it could have been implemented, including simply wrapping the <ctime> stuff, but I wanted to play. It demonstrates some useful STL usage...
• certificate_t
  This is your certificate data. Notice that it is templated on whether or not you wish to print verbosely, since I did not know how you want your data output. (If that is overload for you, then just delete all the "verbose" stuff.) The difference is in the insertion operator beginning on line 255, where you can see the if statement choosing based on what to print. I also had to overload the default and copy constructors for the class in order to be able to change a non-verbose certificate into a verbose one. Again, there are other ways to do this, this is just one.
• expired
  This is a little helper class to select through the list of certificates. See main() for how it is used.

A couple of other things to note:

• Insertion (<<) operators are tricky things to play with. Notice how I output to a stringstream and then send that to the argument stream (assuming no errors). This is generally a good idea...
• Extraction (>>) operators are doubly tricky things to play with. Notice how we do some rudimentary error checking in the extraction operators and set ios::failbit if anything went wrong. There is absolutely no error recovery in this program for invalid stuff -- once something goes wrong it simply stops reading the file.
• The usage() function is designed to complain/help depending on how it is called. Notice how it chooses between standard error and standard out depending on whether it is called because something went wrong or not.
• Your problem statement indicates that five lines should be skipped, but your sample data only has four lines of header data. So I skip up until the first character in the line read was a dash '-', after which the file is read using the fancy extraction operator. Notice that blank lines are tolerated by the operator...
• Next, we sort the data into things past expired, and things not past expired, print, and delete the offending items, if any.
• Finally, we delete everything not soon to be due, and print what remains, if anything.

/*
  Solves http://www.daniweb.com/software-development/cpp/threads/372483

  Copyright 2011 Michael Thomas Greer

  Distributed under the Boost Software License, Version 1.0.
  (See file at http://www.boost.org/LICENSE_1_0.txt )
*/

#include <algorithm>
#include <cctype>
#include <ciso646>
#include <ctime>
#include <fstream>
#include <functional>
#include <iomanip>
#include <iostream>
#include <iterator>
#include <sstream>
#include <string>
#include <vector>
using namespace std;

//----------------------------------------------------------------------------
string titlecase( const string& s )
  {
  string result( s );
  if (result.length())
    {
    transform(
      s.begin(),
      s.end(),
      result.begin(),
      ptr_fun <int, int> ( tolower )
      );
    result[ 0 ] = toupper( s[ 0 ] );
    }
  return result;
  }


//----------------------------------------------------------------------------
struct date_t
//----------------------------------------------------------------------------
  {
  // (table driven data)
  static const char* month_names_abbr[ 12 ];
  static unsigned days_per_month[ 2 ][ 13 ];
  static unsigned yday_per_month[ 2 ][ 13 ];

  unsigned year, month, day;  // 0..N, 1..12, 1..31

  // . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  date_t( unsigned year = 0, unsigned month = 0, unsigned day = 0 ):
    year( year ), month( month ), day( day )
    { }

  // . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  bool is_leap_year() const
    {
    return ((year % 400) == 0) or (((year % 4) == 0) and ((year % 100) != 0));
    }

  // . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  unsigned day_of_year() const
    {
    return yday_per_month[ is_leap_year() ][ month - 1 ] + day;
    }

  // . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  void day_of_year( unsigned d )
    {
    unsigned leap;

    // The following trick allows us to add days...
    while (d > yday_per_month[ leap = is_leap_year() ][ 12 ])
      {
      d    -= yday_per_month[ leap ][ 12 ];
      year += 1;
      }

    if (d > yday_per_month[ leap ][ 12 ]) return;

    month = find_if(
              yday_per_month[ leap ],
              yday_per_month[ leap ] + 12,
              bind2nd( greater_equal <unsigned> (), d )
              )
          - yday_per_month[ leap ];
    day = d - yday_per_month[ leap ][ month - 1 ];
    }

  // . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  bool valid() const
    {
    if ((year < 0) or (month < 1) or (month > 12) or (day < 1))
      return false;
    return day <= days_per_month[ is_leap_year() ][ month ];
    }
  };

//............................................................................
const char* date_t::month_names_abbr[ 12 ] =
  {
  "Jan", "Feb", "Mar", "Apr", "May", "Jun",
  "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
  };

unsigned date_t::days_per_month[ 2 ][ 13 ] =
  {
  { 0, 31, 28, 31,  30,  31,  30,  31,  31,  30,  31,  30,  31 },
  { 0, 31, 29, 31,  30,  31,  30,  31,  31,  30,  31,  30,  31 }
  };

unsigned date_t::yday_per_month[ 2 ][ 13 ] =
  {
  { 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365 },
  { 0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366 }
  };

//............................................................................
istream& operator >> ( istream& ins, date_t& date )
  {
  int    day, year;
  string month;
  ins >> day >> month >> year;
  if (ins)
    {
    month = titlecase( month.substr( 0, 3 ) );
    date.year  = year;
    date.month = find(
                   date_t::month_names_abbr,
                   date_t::month_names_abbr + 12,
                   month
                   )
               - date_t::month_names_abbr + 1;
    date.day   = day;
    if (!date.valid())
      ins.setstate( ios::failbit );
    }
  return ins;
  }

//............................................................................
ostream& operator << ( ostream& outs, const date_t& date )
  {
  ostringstream ss;
  ss << setw( 2 ) << date.day << " " << date_t::month_names_abbr[ date.month - 1 ] << " " << date.year;
  if (!ss) outs.setstate( ss.rdstate() );
  else     outs << ss.str();
  return outs;
  }

//............................................................................
date_t& operator += ( date_t& date, unsigned days )
  {
  date.day_of_year( days + date.day_of_year() );
  return date;
  }

//............................................................................
bool operator == ( const date_t& lhs, const date_t& rhs )
  {
  return (lhs.year == rhs.year) && (lhs.month == rhs.month) && (lhs.day == rhs.day);
  }

//............................................................................
bool operator < ( const date_t& lhs, const date_t& rhs )
  {
  return (lhs.year          != rhs.year)
       ? (lhs.year          <  rhs.year)
       : (lhs.day_of_year() <  rhs.day_of_year());
  }

//............................................................................
bool operator <= ( const date_t& lhs, const date_t& rhs )
  {
  return (lhs == rhs) or (lhs < rhs);
  }

//............................................................................
date_t now()
  {
  date_t    result;
  time_t    t   =  time( NULL );
  struct tm now = *gmtime( &t );

  result.year   = now.tm_year + 1900;
  result.month  = now.tm_mon  + 1;
  result.day    = now.tm_mday;

  return result;
  }


template <bool Verbose = false>
//----------------------------------------------------------------------------
struct certificate_t
//----------------------------------------------------------------------------
  {
  string name;
  string password_policy;
  string type;
  date_t renewal_start_date;
  date_t private_key_expiry;
  string DN;

  certificate_t() { }

  template <bool V>
  certificate_t( const certificate_t <V> & certificate ):
    name               ( certificate.name               ),
    password_policy    ( certificate.password_policy    ),
    type               ( certificate.type               ),
    renewal_start_date ( certificate.renewal_start_date ),
    private_key_expiry ( certificate.private_key_expiry ),
    DN                 ( certificate.DN                 )
    { }
  };

//............................................................................
template <bool Verbose>
istream& operator >> ( istream& ins, certificate_t <Verbose> & certificate )
  {
  string s;

  // Get the next non-empty line
  do getline( ins, s );
  while (ins && s.empty());

  // Remove any trailing whitespace
  s.erase( s.find_last_not_of( " \t\v\f\r\n" ) + 1 );

  istringstream ss( s );
  ss >> s;
  ss >> certificate.name
     >> certificate.password_policy
     >> certificate.type
     >> certificate.renewal_start_date
     >> certificate.private_key_expiry
     >> ws;
  getline( ss, certificate.DN );

  certificate.name = s + " " + certificate.name;

  if (ss.fail())
    ins.setstate( ss.rdstate() );

  // Do some error checking here on password_policy, type, and DN //

  return ins;
  }

//............................................................................
template <bool Verbose>
ostream& operator << ( ostream& outs, const certificate_t <Verbose> & certificate )
  {
  ostringstream ss;
  ss << left;
  if (Verbose)
    ss << setw( 16 ) << certificate.name               << " "
       << setw( 11 ) << certificate.password_policy    << " "
       << setw( 12 ) << certificate.type               << " "
       << setw( 15 ) << certificate.renewal_start_date << " "
       << setw( 15 ) << certificate.private_key_expiry << " "
       <<               certificate.DN;
  else
    ss << setw( 16 ) << certificate.name               << " "
       << setw( 15 ) << certificate.renewal_start_date << " "
       << setw( 15 ) << certificate.private_key_expiry;
  if (!ss) outs.setstate( ss.rdstate() );
  else     outs << ss.str();
  return outs;
  }


//----------------------------------------------------------------------------
struct expired
//----------------------------------------------------------------------------
  {
  date_t when;

  explicit expired( const date_t& when ):
    when( when )
    { }

  template <bool Verbose>
  bool operator () ( const certificate_t <Verbose> & certificate ) const
    {
    return not(    (certificate.renewal_start_date <= when)
                or (certificate.private_key_expiry <= when) );
    }
  };


//----------------------------------------------------------------------------
int usage( const char* arg0, int exitcode, const char* message = "" )
  {
  (exitcode ? cerr : cout) << message << (message ? "\n" : "") <<
    "usage:\n"
    "  " << arg0 << " --help\n"
    "  " << arg0 << " /?\n"
    "    Display this help.\n\n"

    "  " << arg0 << " [--verbose] DAYS FILENAME\n"
    "  " << arg0 << " [/v]        DAYS FILENAME\n"
    "    Search the file for certificates expiring in DAYS days or less.\n"
    "    If the VERBOSE option (\"--verbose\" or \"/v\") is specified, then\n"
    "    all data about the matching entry is displayed, otherwise only the\n"
    "    entry name is displayed.\n"
    "    DAYS must be a non-negative whole number.\n\n";
  return exitcode;
  }

//----------------------------------------------------------------------------
int getdays( const char* days )
  {
  int result = -1;
  istringstream ss( days );
  ss >> result;
  return result;
  }

//----------------------------------------------------------------------------
bool ishelp( const string& arg1 )
  {
  return (arg1 == "--help") or (arg1 == "/?");
  }

//----------------------------------------------------------------------------
bool isverbose( const string& arg1 )
  {
  return (arg1 == "--verbose") or (arg1 == "/v");
  }

//----------------------------------------------------------------------------
int main( int argc, char** argv )
  {
  // Parse command arguments .................................................
  if ((argc == 1) or ishelp( argv[ 1 ] ))
    return usage( argv[ 0 ], 0 );

  bool verbose = isverbose( argv[ 1 ] );
  if (argc != (verbose ? 4 : 3))
    return usage( argv[ 0 ], 1, "Incorrect arguments" );

  int days = getdays( argv[ verbose ? 2 : 1 ] );
  if (days < 0)
    return usage( argv[ 0 ], 1, "Invalid DAYS" );

  ifstream f( argv[ verbose ? 3 : 2 ] );
  if (!f)
    return usage( argv[ 0 ], 1, "Invalid FILENAME" );

  date_t today = now();
  date_t tomorrow = today; tomorrow += days;

  // Skip the non-data lines .................................................
  // (up to and including the first line starting with a "-")
  {
    string s;
    do    getline( f, s );
    while (s.substr( 0, 1 ) != "-");
  }

  vector <certificate_t <> > certificates;
  copy(
    istream_iterator <certificate_t <> > ( f ),
    istream_iterator <certificate_t <> > (),
    back_inserter( certificates )
    );

  cout << "There are " << certificates.size() << " certificates.\n\n";

  // Find and display all items past due .....................................
  // (Expired items are moved to the END of the vector)
  vector <certificate_t <> > ::iterator
  certificates_past_due_begin
    = stable_partition(
        certificates.begin(),
        certificates.end(),
        expired( today )
        );
  if (certificates_past_due_begin != certificates.end())
    {
    cout << "Items PAST DUE\n  ";
    if (verbose)
      copy(
        certificates_past_due_begin,
        certificates.end(),
        ostream_iterator <certificate_t <true> > ( cout, "\n  " )
        );
    else
      copy(
        certificates_past_due_begin,
        certificates.end(),
        ostream_iterator <certificate_t <false> > ( cout, "\n  " )
        );
    certificates.erase( certificates_past_due_begin, certificates.end() );
    }
  else
    cout << "No items past due\n";

  // Find and display all items due "tomorrow" (today + DAYS) ................
  // (Erase all items not soon to expire)
  certificates.erase(
    remove_if(
      certificates.begin(),
      certificates.end(),
      expired( tomorrow )
      ),
    certificates.end()
    );
  if (certificates.size())
    {
    cout << "\nItems due to expire in " << days << " DAYS\n  ";
    if (verbose)
      copy(
        certificates.begin(),
        certificates.end(),
        ostream_iterator <certificate_t <true> > ( cout, "\n  " )
        );
    else
      copy(
        certificates.begin(),
        certificates.end(),
        ostream_iterator <certificate_t <false> > ( cout, "\n  " )
        );
    }
  else
    cout << "\nNo items due to expire in " << days << " days\n";

  return 0;
  }

Whew. Hope this helps.

[mhaber]

Wow..you must have been incredibly bored! Anyway, thanks.
At first the code looked really complicated but it's starting to come together..
I'm just having some troubles with:

{
string s;
do getline( f, s );
while (s.substr( 0, 1 ) != "-");
}

vector <certificate_t <> > certificates;
copy(
istream_iterator <certificate_t <> > ( f ),
istream_iterator <certificate_t <> > (),
back_inserter( certificates )
);

do-while works fine by skipping the first lines until the multiple dashes, but the copy() doesn't fill the vector as it should:

root@UBILUSNAGPRD01:~# ./parser2 365 /mnt/SNLDrive/certlistout.txt
There are 0 certificates.

[Duoas]

I tested pretty carefully on the example data you gave me.
What does the header for your data actually look like?

The loop I gave you only stops when it finds a line whose very first character is a dash '-' . If it doesn't find such a line, then it will scan through the entire file and your vector of certificates will never get filled.

If that is not the case, post back with the actual header for your file and any changes to the code you made.

Hope this helps.

[mhaber]

The header and the file format is exactly the same as I posted on the forum.

And I got the part about the do-while, which works great (I printed out the contents of the string s and it definitely contains the header), but it looks like the part where the vector gets filled is where the trouble starts.