Search This Blog

Wednesday, February 27, 2013

W3C - LinkChecker - post processing with Awk

The W3C link checker program is very useful for checking large numbers of html files.
I have used the Perl CPAN implementation
One way to check a lot of pages from a command prompt is a script like this:

cd /home/webserver/pages
for page in *.htm *.html; do
/usr/bin/linkcheck -s ${baseurl}${page} >> outputfile

This may take a while to run the checking process is very thorough and the reports are quite verbose.
A common requirement is to just check for 404 (bad link) errors. To only report these I filtered the output file through an awk script:

# from FTP::webx-johnr\/home/johnr/librarycheck|linkfilter.awk
BEGIN { url = ""; }
/^Processing/  { url=$2;
/^http:/ { link=$1;  }
/^ Lines: / {lines = $2 $3; }
/^  Code: 404 Not Found/  { 
 if (! errorCount) printf "\n\nCompany page: %s\n", url;
printf "link: %s lines %s; %s\n", link, lines, $0; 
END {}

This only lists pages where 404 errors have occurred, ok pages or other 'error's such as redirections or ignored links are not listed.

Thursday, February 21, 2013

Symfony 2 Command Line

Symfony 2 Command Line

This well documented in Git/Packagist and also here in in the Symfony 2 documentation.
This post will go into more detail re running classes from the command line, but first a bit more information, re the example class that extends the Command class:

namespace Acme\DemoBundle\Command;

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;

class GreetCommand extends Command
    protected function configure()
            ->setDescription('Greet someone')
                'Who do you want to greet?'
               'If set, the task will yell in uppercase letters'

    protected function execute(InputInterface $input, OutputInterface $output)
        $name = $input->getArgument('name');
        if ($name) {
            $text = 'Hello '.$name;
        } else {
            $text = 'Hello';

        if ($input->getOption('yell')) {
            $text = strtoupper($text);


The parts in bold - specifically:

  • The end of the class name GreetCommand,
  • The parent folder - Command (and last item in the namespace path)
Must both be Command for the classes to be recognised by the app/console script without any further configuration.
Also the string parameter in the setName call should be checked and if necessary edited.
I was able to get custom commands working in Symfony 2.1 by:
  1. Installing the component using Composer.
  2. Creating the above php file - editing the value above as needed.
$ app/console --list
Symfony version 2.1.8-DEV - app/dev/debug
  test:greet                            Greet someone
  test:update                           Update someone


Wednesday, February 13, 2013

Symfony project sfservermon post 4 - edit forms.

This is post 4 in the series and covers the edit forms. The server and logs listing pages are discussed in post 3  here

The existing phpservermon application has edit forms for the server list, users and configuration data.

As the forms (with the exception of the configuration data) map well to the Symfony forms framework, the forms for sfservermon were generated using the Sensio Generator Bundle

Using the Sensio Generator Bundle

The servers edit forms were generated:

$ app/console generate:doctrine:crud \
> --entity=JMPRServerMonBundle:MonitorServers

  Welcome to the Doctrine2 CRUD generator 

This command helps you generate CRUD controllers and templates.

First, you need to give the entity for which you want to generate a CRUD.
You can give an entity that does not exist yet and the wizard will help
you defining it.

You must use the shortcut notation like AcmeBlogBundle:Post.

The Entity shortcut name [JMPRServerMonBundle:MonitorServers]:

By default, the generator creates two actions: list and show.
You can also ask it to generate "write" actions: new, update, and delete.

Do you want to generate the "write" actions [no]? yes

Determine the format to use for the generated CRUD.

Configuration format (yml, xml, php, or annotation) [annotation]: yml

Determine the routes prefix (all the routes will be "mounted" under this
prefix: /prefix/, /prefix/new, ...).

Routes prefix [/monitorservers]: /servers

  Summary before generation 

You are going to generate a CRUD controller for "JMPRServerMonBundle:MonitorServers"
using the "yml" format.

Do you confirm generation [yes]?

  CRUD generation 

Generating the CRUD code: OK
Generating the Form code: OK
Confirm automatic update of the Routing [yes]?
Importing the CRUD routes: FAILED

  The command was not able to configure everything automatically. 
  You must do the following changes manually.                     

- Import the bundle's routing resource in the bundle routing file

        resource: "@JMPRServerMonBundle/Resources/config/routing/monitorservers.yml"
        prefix:   /servers

This does the following:
  • Creates a routing file in <bundle>/Resources/config/routing/monitorservers.yml
  • Creates a controller class MonitorServersController in <bundle>/Controllers/MonitorServersController.php
  • Creates a form type class MonitorServersType in <bundle>/Form/MonitorServersType.php
  • Created index, edit, new and show .twig.html view in <bundle>//Resources/views/MonitorServers
Once the servers.html.twig template is edited to add the routes, the form can be navigated to:

{% block pageContent %}
<div class="message"><a href="{{ path('servers_new')}}">Add new?</a></div><br/>
<table cellpadding="0" cellspacing="0">


   <a href="{{ path('servers_edit', { 'id': })}}"><img src="/img/edit.png" alt="edit" title="Edit Server" /></a>


   <a href="javascript:sm_delete('{{ path('servers_edit', { 'id': })}}', 'servers');"><img src="/img/delete.png" alt="delete" title="Delete Server" /></a>

The generated MonitorServers/edit.html.twig template is improved:
  • Add twig reference to base.html.twig
  • Add block references.
  • Add a stylesheet editrecords.css.
  • Add reference to editrecords styles .
{# src/JMPR/ServerMonBundle/Resources/views/Default/index.html.twig #}
{% extends '::base.html.twig' %}
{% block title %}{{title}}{% endblock %}
{% block stylesheets %}
<link type="text/css" href="/css/editrecords.css" rel="stylesheet" />
{% endblock %}
{% block pageContent %}
<h1>MonitorServers edit</h1>

<form action="{{ path('servers_update', { 'id': }) }}" method="post" {{ form_enctype(edit_form) }}>
    <input type="hidden" name="_method" value="PUT" />
    {{ form_widget(edit_form) }}
        <button type="submit">Edit</button>

<ul class="record_actions">
        <a class="record_actions" href="{{ path('servers') }}">
            Back to the list
        <form action="{{ path('servers_delete', { 'id': }) }}" method="post">
            <input type="hidden" name="_method" value="DELETE" />
            {{ form_widget(delete_form) }}
            <button type="submit">Delete</button>

    <input type="hidden" name="_method" value="PUT" />
    {{ form_widget(edit_form) }}
        <button type="submit">Edit</button>

{% endblock %}

The server edit form can be viewed:

This works, but has issues. We need to:
  • Change the port field,
  • Change to type field to a select menu.
  • Remove the un-needed fields.
Also we should add validation using the Symfony validation scheme.

New Record Defaults:

For compatibility with PhpServermon, the following fields should be set to these defaults in the newAction in the MonitorServersController class.

    public function newAction()
        $entity = new MonitorServers();
        $form   = $this->createForm(new MonitorServersType(), $entity);

Also in the updateAction method, this will set the Error to a non null value before it is saved.
    public function createAction(Request $request)
        $entity  = new MonitorServers();
        $form = $this->createForm(new MonitorServersType(), $entity);

Initial Edit.

This is done by changing add calls from the BuildForm method in MonitorServersType. See the forms reference.
    public function buildForm(FormBuilderInterface $builder, array $options)




            ->add('ip','url', array('label'=>'Domain/IP', 'attr'=>array('size'=>60)))


            ->add('type', 'choice', array('choices'=>array('service'=>'Service', 'host'=>'Host', 'website'=>'Website')))
            ->add('active', 'choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No')))
            ->add('email', 'choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No')))
            ->add('sms', 'choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No')))

In particular:
  • Un-needed fields are removed,
  • Uhe order is changed,
  • Ip/address field has size set to 60.
  • A label is added for the ip field,
  • Choice fields and choices arrays are set for the drop down menus.
Now the form looks like this:

This is better, but the rendering of the form gives us limited options.
An alternative is to render each form in the template - for exmaple in table, but I was keen to use the builder method in the Type class and the  form_widget(edit_form) statement

So if we want to use the simple form_widget statement, we need to modify the css used.
To set the sytle class, add a class item to the attr array and add a label_attr item - with a class item.
            ->add('label', null, array('attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))

This can be done by setting style classes for the form labels and the field, in editrecords.css:
width: 120px;
float: left;

width: 240px;
float: right;
Disclaimer - IAMAHD (I Am Not A Html Designer), it is quite likely that there is better css that this.
The builder call becomes:
            ->add('label',null, array('attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('ip','url',array('label'=>'Domain/IP', 'attr'=>array('size'=>60, 'class' => 'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('port',null, array('attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('status',null, array('attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('type','choice', array('choices'=>array('service'=>'Service', 'host'=>'Host', 'website'=>'Website'), 'attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('active', 'choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No'),'attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('email','choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No'),'attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))
            ->add('sms','choice', array('choices'=>array('yes'=>'Yes', 'no'=>'No'),'attr' => array('class' =>'edit_field'), 'label_attr' => array('class' => 'edit_label')))

And the form:

The layout can be improved but it ok for an initial release.

Handling Deletes.

Servermon uses http get rather than post to request a delete, so the requirement was removed from the route:

    pattern:  /{id}/delete
    defaults: { _controller: "JMPRServerMonBundle:MonitorServers:delete" }
    requirements: {  }
and the controller changed:
    public function deleteAction(Request $request, $id)
        $em = $this->getDoctrine()->getManager();
        $entity = $em->getRepository('JMPRServerMonBundle:MonitorServers')->find($id);

        if (!$entity) {
            throw $this->createNotFoundException('Unable to find MonitorServers entity.');


        return $this->redirect($this->generateUrl('server_mon_homepage'));

Symfony Project sfservermon - Post 2 Data (awk digression)

This is the second post in the series, the first is here

The intention of this project is work with existing data - the schema of the database will be unchanged.
For development I wanted some test data - also the configuration data was required.

The aim of this post is to have some development data which can be loaded as required into the database while developing the system.
The format of the data files is yml, loaded using the excellent bundle from Khepin - , I discuss using his bundle here.

This post - which is a digression into the world of sed and awk, discusses generating yml fixtures from csv files, however as I only wanted a sample 1 - 5 rows, I decided to see if I would read parse the data from MySQL statements, particular those formatted with the \G separator, an example:

mysql> select * from monitor_log limit 3\G
*************************** 1. row ***************************
   log_id: 1
server_id: 4
     type: status
  message: Server 'press release' is DOWN: ip=, port=80. Error=404 Not Found
 datetime: 2012-11-15 15:26:07
*************************** 2. row ***************************
   log_id: 2
server_id: 4
     type: status
  message: Server 'press release' is RUNNING: ip=, port=80
 datetime: 2012-11-15 15:30:21

Note that the table name has a space after it, so if not using limit or order by:

mysql> select * from monitor_config \G

To me the data looks very much like the needed yml.
Awk is the perfect tool for the editing required.

Strictly speaking this development was not required at all,  I could export and reimport the database and only keep the required rows.
This post is a (perhaps) useful documentation of how awk can be used.

The first version:

BEGIN { } 
/^mysql> select/ { table = $5;sub(/\\G/, "", table);
printf "model: %s\n", table;
printf "persistence: orm\n"
printf "fixtures:\n";
        primaryKey= substr(table "_id", prefixLen+1) ":";
/\*+.* row/ { printf "  %s_%0.4d:\n",table,++fixture ; }
/^[^\*]*: NULL$/    { printf "    %s ~\n",$1; next; }

/^[^\*]*: .*/    {s = ""; for (i = 2; i <= NF; i++) s = s $i " "; 
                        printf "    %s %s\n",$1,s; }

END { }
  1. The begin and empty actions are currently empty.
  2. The table select statement is matched and parsed to get the table name.
  3. The model name and header lines are printed.
  4. The end of each row is matched "**** row" and then printed. All of the subsequence fields are concatenated and printed. Printf is used to get the space indented formatting required.
  5. NULL fields are matched and a tilde '~' printed.

This generates this output:

johnreidy$ awk -f datayml.awk.1 < log.list 

model: monitor_log
persistence: orm

    log_id: 1 
    server_id: 4 
    type: status 
    message: Server 'press release' is DOWN: ip=, port=80. Error=404 Not Found 
    datetime: 2012-11-15 15:26:07 
    log_id: 2 
    server_id: 4 
    type: status 
    message: Server 'press release' is RUNNING: ip=, port=80 
    datetime: 2012-11-15 15:30:21 

As a first pass this is not bad.
Second Pass - remove primary keys and format dates.
The import files cannot have a primary key fields - log_id, doctrine will report an error if they are present, this makes sense as the entities themselves do not have s setId method.
The problem is that the id fields are not id and not even <table_name>_id, e.g. monitor_log_id.
These tables all have a prefix of monitor_. This is passed to the awk script as an environment variable - prefix.
BEGIN { prefix=ENVIRON["prefix"]== "" ? "": ENVIRON["prefix"] "_"; prefixLen= length(prefix); }
/^mysql> select/ { table = $5;sub(/\\G/, "", table);
        printf "model: %s\n", table;
        printf "persistence: orm\n"
        printf "fixtures:\n";
        primaryKey= substr(table "_id", prefixLen+1) ":";
/\*+.* row/ { printf "  %s_%0.4d:\n",table,++fixture ; }
$1 == primaryKey { next; }
/^[^\*]*: NULL$/    { printf "    %s ~\n",$1; next; }
/^[^\*]*: [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].*/ {
                s = ""; for (i = 2; i < NF; i++) s = s $i " "; s = s $i;
                printf "    %s \"%s\"\n",$1,s; next; }
/^[^\*]*: .*/    {s = ""; for (i = 2; i < NF; i++) s = s $i " "; s = s $i;
                        printf "    %s %s\n",$1,s; }
END { } 
This now generates:
model: monitor_log
persistence: orm
    server_id: 4 
    type: status 
    message: Server 'press release' is DOWN: ip=, port=80. Error=404 Not Found 
    datetime: "2012-11-15 15:26:07"
    server_id: 4 
    type: status 
    message: Server 'press release' is RUNNING: ip=, port=80 
    datetime: "2012-11-15 15:30:21"

I also edited the loop to not append the space after the last field.
Also quotes are put around the date fields.

Plural Table Names.

This is good but there was a problem with another table - monitor_users:
model: monitor_users
persistence: orm
    user_id: 1 
    server_id: 4,3,5,6,9,10,15 
    name: John Reidy 
    mobile: 61415552311 
    user_id: 2 
    server_id: 4,3,5,6,9,10 
    name: Fred James 
    mobile: 61912555231 
The table name is a plural - users but the primary key is user_id.
But this stage I wasn't sure if using awk was a good approach but I pushed on.
The solution is to add an extra statement:
/^mysql> select/ { table = $5;sub(/\\G/, "", table);
printf "model: %s\n", table;
printf "persistence: orm\n"
printf "fixtures:\n";
        sub(/s$/, "", table);
        primaryKey= substr(table "_id", prefixLen+1) ":";

One possibility is to add an extra match which would be more awkish.
One more hack to the awk was needed, the data has not null fields, but with a single space in them, this was not being picked up by awk I think it is not handled by the tee statement (used to capture the output from mysql).
/^[^\*]*: $/    { printf "    %s \" \"\n",$1; next; }

This prints a " " sequence if there is no data.

Adding the Correct Model Statement.

This is now pretty close. However the model statement needs correction, it should read:
model: JMPR\ServerMonBundle\Entity\MonitorUsers
rather than
model: monitor_user

As this requires a change to title case for the entities, sed appears to be a better tool.
So I run the generated file through a sed postprocessor script:

sed  -e '/^model/{s/[^ _\\]\+/\L\u&/g}' \
     -e '/^Model/{s/_//g}' \
     -e "/^Model/{s#Model: #model: ${model}#}" 

This does the following:
  1. Change the table name to title case.
  2. Remove the underscores.
  3. Revert the model tag back to model from Model (as changed in step 1) and prepend the contends of the model environment variable.
Wrapped in a script this becomes:

export model='JMPR\\ServerMonBundle\\Entity\\'
export prefix='monitor'

cat $sourcepath/$file |awk -f $scriptdir/data4.awk  | \
sed  -e '/^model/{s/[^ _\\]\+/\L\u&/g}' \
     -e '/^Model/{s/_//g}' \
     -e "/^Model/{s#Model: #model: ${model}#}" \
> ${destpath}/${file}
echo "output to: ${destpath}/${file}"

With the data load, the next post covers the controllers and templates to view the server list and logs.

Symfony Project sfservermon - Post 1 - Step 1 Entities

As an example Symfony 2 project, this series of blog posts details the migration of an existing php website project to the Symfony 2 framework.
The project the subject of this series of posts is php server mon - see Php Server Monitor on sourceforge.
Php Server Monitor is an monitoring system developed by Pep, it was chose for this project as an example because:

  • The application developed by Pep is well written and great at what is does.
  • It is a useful example - different from other examples - but not too complex.
  • It provides a useful example of migrating an existing application to Symphony.


The purpose of migrating it to Symfony 2 is not to improve the site or the code as that is not needed but to document migrating an existing project to Symfony and then extending it to support mobile phone browsers through JQuery mobile.

All existing functionality will be maintained, this includes:

  • monitoring hosts (by ping), websites (by a http get) and services (by a telnet connection).
  • logging the results.
  • providing alerts - webpage display, emails and alerts.
  • database schema is unchanged, so the new project could be a direct upgrade of the existing application.
The initial implementation will support English, however it will then be extended with language support - as the existing project does.

Later in the series I do have a couple of posts about some added features.

Also, the code will be edited as per the bundle best practice guidelines.

Step 1 Creation of ORM files and Entities.

The existing schemas has the following tables:
  • monitor_config - primary key (config_id) - application configuration data.
  • monitor_servers - primary key (server_id) - list of monitoring targets (servers).
  • monitor_log - primary key (log_id) - has many to one relation to monitor_servers.
  • monitor_users - primary key (user_id) - has comma seperated list of servers to be alerted.

This cookbook article was followed to developed the orm files, then some edits were required:

1a Mysql Enum support.

This post was followed to add the following enum:
  `type` enum('service','website','host') NOT NULL DEFAULT 'service' COMMENT '(DC2Type:enumservermonitor)',

1b. Keyword Column support.

The monitor_config table has the SQL:

CREATE TABLE `monitor_config` (
  `config_id` int(11) NOT NULL AUTO_INCREMENT,
  `key` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `value` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`config_id`)

The key column is a keyword and needs to be escaped.
Add this statement to the orm file as generated above:

      column: `key`
      type: string
      length: 255
      fixed: false
      nullable: false
      column: `value`
      type: string
      length: 255
      fixed: false

The same was done for the value column, though not necessary.

1c. Renaming Mapped Column Names.

The primary key fields are non standard, examples are
Servers: server_id, Users: user_id rather than id.
You can renamed the field names in the yml files which will keep the database column name unchanged, but rename the Entity field to id.

  type: entity
  table: monitor_config
      id: true
      type: integer
      unsigned: false
      nullable: false
      column: config_id
        strategy: IDENTITY


  type: entity
  table: monitor_config
      id: true
      type: integer
      unsigned: false
      nullable: false
      column: config_id
        strategy: IDENTITY

Note the column field (config_id)

1d. Relations support.

The orm generation doesn't include support for the mapping (relations between the tables)
These where added to the orm files:

      targetEntity: MonitorLog
      mappedBy: log
Monitor log:

       targetEntity: MonitorServers
       inversedBy: server
         name: server_id
         referencedColumnName: server_id

As no standard primary key fields are used (log_id rather than id in monitor_log), the join column must be specified.

The next post discusses loading test data.

Monday, February 4, 2013

Symfony 2 / Doctrine Support Enum Data types.

These posts were followed to add enum support to this Symfony 2 project:,
This post is to document the steps done to implement it, under Symfony 2.2.0

1. Add dbal configration in app/config/config.yml

        driver:   %database_driver%
        host:     %database_host%
        port:     %database_port%
        dbname:   %database_name%
        user:     %database_user%
        password: %database_password%
        charset:  UTF8
            enum: string
            enumservermonitor: JMPR\ServerMonBundle\Type\EnumServerMonitorType
            enumserverstatus: JMPR\ServerMonBundle\Type\EnumServerStatusType
            enumalertaction: JMPR\ServerMonBundle\Type\EnumAlertActionType

2. Create the column class types.

As specified in the config.yml file the Enum...Type classes are created.

The EnumServerMonitor type is below.
It does the following:
  • defines a classname and the enum values (service/website/host).
  • specifies the sql declaration.
  • defines 2 methods to convert the value to php from the database and from php to the database value.

namespace JMPR\ServerMonBundle\Type;

use Doctrine\DBAL\Types\Type;
use Doctrine\DBAL\Platforms\AbstractPlatform;

class EnumServerMonitorType extends Type
    const ENUM_SERVERMONITOR = 'enumservermonitor';
    const VALUE_SERVICE = 'service';
    const VALUE_WEBSITE = 'website';
    const VALUE_HOST = 'host';

    public function getSqlDeclaration(array $fieldDeclaration, AbstractPlatform $platform)
        return "enum('service','website','host') NOT NULL default 'service' COMMENT '(DC2Type:enumservermonitor)'";

    public function convertToPHPValue($value, AbstractPlatform $platform)
        return $value;

    public function convertToDatabaseValue($value, AbstractPlatform $platform)
        if (!in_array($value, array(self::VALUE_SERVICE, self::VALUE_WEBSITE, self::VALUE_HOST))) {
            throw new \InvalidArgumentException("Invalid monitor type");
        return $value;

    public function getName()
        return self::ENUM_SERVERMONITOR;