Loopback model migration using postgresql database

08 Jan 2017

I have been playing with Loopback, initially I was just declaring models and use in memory, but now I got to a point where I need to have a persistent database.

I couldn’t find how to keep my database synced with my models easily. I’m not sure if I’m not that familiar with Loopback yet, or if their documentation is not clear enough.

To create a script to sync your models with your database you can create a file under bin/ called autoupdate.js and add the following:

var path = require('path');

var app = require(path.resolve(__dirname, '../server/server'));
var ds = app.datasources.db;
ds.autoupdate(function(err) {
  if (err) throw err;
  ds.disconnect();
});

The code is pretty simple, it will fetch the app from server.js, grab the datasource and run the autoupate command. You could use automigrate, but this one will clean the database every time, so pay attention on this.

I think this will work for most of datasources, but if it doesn’t work for yours, drop me a line. I can try to help :D

Matheus

PS: Loopback will not create migrations and do a proper job as Django, sometimes you can get to weird states, it seems Loopback works better with NoSQL databases.

Comment

Django Storages with Boto3 and additional Metadata only for Media

29 Dec 2016

I have a personal project which I’m using python with Django and django-storages to upload my static and media files to Amazon S3, because my media files have UUID and they’re not editable on my system I wanted to have a long expiration time on it, so I could save some bandwidth but I didn’t want this on the static files which are updated more regularly when I’m updating the system.

Most of resources refer to AWS_HEADERS but it didn’t work for me. It seems it is only for boto (not boto3) after looking into boto3 source code I discovered AWS_S3_OBJECT_PARAMETERS which works for boto3, but this is a system-wide setting, so I had to extend S3Boto3Storage.

So the code that solved my problem was:

class MediaRootS3Boto3Storage(S3Boto3Storage):
    location = 'media'
    object_parameters = {
        'CacheControl': 'max-age=604800'
    }

If you’re using boto (not boto3) and you want to have specific parameters only for Media classes you could use

class MediaRootS3Boto3Storage(S3BotoStorage):
    location = 'media'
    headers = {
        'CacheControl': 'max-age=604800'
    }

You also need to update your django-storages settings, pay attention to the class name, on boto 3 it is S3Boto3Storage on boto it doesn’t has the 3 after Boto.

DEFAULT_FILE_STORAGE = 'package.module.MediaRootS3Boto3Storage'

Very simple tip, but it took a while to find out how it works

Matheus

Comment

Nginx redirect on failure

22 Dec 2016

As a few of you probably noticed, recently I have decided to update my really old wordpress blog from PHP4~5 to a most recent one. Leaving a shared host and going to heroku, which later became Amazon EC2.

I had to decide if I would keep Wordpress, or change to a different technology as Jekyll? Or what? I have thought a lot about this and in the end I decided to use Jekyll to be honest, why? Because using something new will motivate me to study, play with something new and work more.

Have decided to work with Jekyll, I had to think about my domain, because I didn’t want to break my old wordpress blog, I want to keep it alive as a record and keep it for SEO points, but how to keep both living together on an awesome way?

I thought the ideal would be to have something that tries to access the new website and if it is not found it should redirect to the old wordpress website. But how to redirect to the old blog only when a page is not found and complying with the http status code (ie: redirecting with 301).

After some documentation reading on nginx I found you can try to proxy to a server and if it fails redirect to a new one, it seems the ideal solution for now.

I have a nginx configuration file with multiple servers, first I have a nginx wordpress configuration, this server just adds PHP-FPM to process PHP files basically with my own custom domain.

server {
   listen 80;
   server_name wordpress.matbra.com;

    location / {
        root   /var/www/wordpress/live;
        index  index.php index.html index.htm;
        try_files $uri $uri/ /index.php?$uri$args;
    }

    location ~ \.php$ {
        root /var/www/wordpress/live;
        fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }
}

Build Jekyll as production after push

12 Dec 2016

If you want to build your Jekyll blog on your own server after a git push you can use git hooks. To do it, you can extend the Deploy after git push and add this tree lines (after rm -rf), to install dependencies and to build it as production environment.

	cd $LIVE_PATH
	bundle install
	JEKYLL_ENV=production jekyll build

Matheus

Comment

Force www on Jekyll website using Javascript

11 Dec 2016

I wanted to force using my jekyll website to have the “www” prefix and because my Jekyll doesn’t have a back-end I couldn’t do it on the server, so I needed to use Javascript or Meta tags. A few people says Google Search engine handles meta refresh as 301/302 so it would be better to go with this approach from a SEO perspective.

If you want to force www prefix on your website using javascript, you can use this snippet:

<script>
if (window.location.hostname.indexOf("www") != 0) {
	window.location = window.location.protocol + "//www." + window.location.hostname + window.location.pathname;
}
</script>

I have created a _include/force_www.html file and I’m using jekyll.environment to load it, so I’m only loading it on production.

Matheus

Comment

Set env var to PHP-FPM

10 Dec 2016

After installing nginx and php, I wanted to use environment vars inside PHP 7 so I don’t need to save configuration to my repo.

Usually when using environment vars the ideal is to set it without having it saved in a file but on this case it was easier to.

If you want to add environment variables to your PHP-FPM you can edit /etc/php-fpm.d/www.conf (I’m doing it on Amazon Linux and PHP 7.0)

There is a flag clear_env = no where you’re able to set if php-fpm will receive a clean environment or not. I decided to leave it as the default value and but setting my vars as

env[WP_SECURE_AUTH_KEY] = "some-value"
env[WP_NONCE_KEY] = "nonce-key"

After this I restarted my nginx and php-fpm.

sudo service nginx restart
sudo service php-fpm restart

Matheus

Comment

Deploy after push to your own git

09 Dec 2016

I have explained how to push your code to your own git server and after this you may want to execute some especific functions, in my specific case I wanted my code to be builded and to release a new version, so I used post-receive hook from my repo.

Oh, it also handle multiple versions keeping the last 3 versions of the release. To do this it uses your DEPLOY_PATH and create a new folder sources on it, which will have your versions and a live folder which is a symlink to the version which is running.

Vars:

REPO_PATH = Path to your git folder
DEPLOY_PATH = Path to your destiny folder
DEPLOY_BRANCH = Branch you want to deploy

#!/bin/bash
REPO_PATH=/home/someuser/test.git
DEPLOY_PATH=/var/www/
DEPLOY_BRANCH="master"

echo "REPO_PATH=$REPO_PATH"
echo "DEPLOY_PATH=$DEPLOY_PATH"

while read oldrev newrev refname
do
    branch=$(git rev-parse --symbolic --abbrev-ref $refname)
    if [ $DEPLOY_BRANCH == "$branch" ]; then
        TIMESTAMP=$(date +%Y%m%d%H%M%S)
        VERSION_PATH=$DEPLOY_PATH/sources/$TIMESTAMP
        LIVE_PATH=$DEPLOY_PATH/live
        echo "TIMESTAMP=$TIMESTAMP"
        echo "VERSION_PATH=$VERSION_PATH"
        echo "LIVE_PATH=$LIVE_PATH"

        mkdir -p $VERSION_PATH
        mkdir -p $VERSION_PATH/sources

        git --work-tree=$VERSION_PATH --git-dir=$REPO_PATH checkout -f $DEPLOY_BRANCH
        # Remove git files
        rm -rf $VERSION_PATH/.git
        rm -rf $LIVE_PATH
        ln -s $VERSION_PATH $LIVE_PATH


        # Delete old folder keeping the 3 most recent ones, which aren't the current live one, / (root, security measure, different from your source folder)
        rm -rf $(ls -1dt $(find -L $DEPLOY_PATH/sources/ -maxdepth 1 -type d ! -samefile / ! -samefile $DEPLOY_PATH/sources/ ! -samefile $LIVE_PATH -print) | tail -n+3)
    fi
done

If you have any question, let me know. Matheus

Comment

Pushing to your own remote git

08 Dec 2016

I’m creating a new server as you can notice and I would like to push directly to my git (hosted on my own server), so I could release a new version with a simple git push myserver branch.

If you want to achieve this as well you can connect to your remote ssh and execute

$ mkdir test.git
$ cd git
$ git --bare init

You will need to know the full path of your git folder to add to as a remote on your local, to check the full path run pwd. Back to your local machine add your remote server.

git remote add my_server ssh:[email protected]:/replace/with/pwd/test.git

After this you can use git push my_server branch to push to it.

Matheus

Comment

Install Nginx, PHP on Amazon Linux

07 Dec 2016

I’m migrating my blog and a few other stuff I have running to Amazon infrastructure. I needed an Amazon EC2 instance with PHP support and able to connect to a MySQL.

Steps:

yum update
yum install nginx
yum install php70 php70-fpm php70-mysqlnd
Edit /etc/nginx/conf.d/virtual.conf

server {
    listen       3000;

    location / {
        root   /var/www/;
        index  index.php index.html index.htm;
    }

    location ~ \.php$ {
        root /var/www/;
        fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }
}

Edit the following properties of: /etc/php-fpm-7.0.d/www.conf

user = nginx
group = nginx

listen = /var/run/php-fpm/php-fpm.sock

listen.owner = nginx
listen.group = nginx
listen.mode = 0660

Create a php file on /var/www/

<?php
phpinfo();
?>

Access http://SERVER_IP:3000

You will need your security group for your ec2 instance to have port 3000 opened.

If you want to add them to auto start:

sudo chkconfig nginx on
sudo chkconfig php-fpm on

If you want to restart this services:

sudo service nginx restart
sudo service php-fpm restart

Matheus

Comment

Migrate old Wordpress to Heroku, Amazon RDS and S3.

03 Dec 2016

After a few good years with my blog out of date, I decided to start to write again and to migrate it to Heroku since his server was with a really old stack. I decided to use Heroku, Amazon RDS as Database service and S3 as file storage (for uploaded files)

Steps:

Disable all Wordpress’ extensions
Do a full backup (Wordpress, Database, Uploads, etc)
Really, do a backup!
Create a git repository
Add Wordpress code to your git
1. If you want to update your Wordpress add latest Wordpress version
  1. Don’t add your private configs (wp-config.php)!
  2. Don’t add uploads folder
  3. Add your plugins
  4. Add your theme
2. If you want to keep your Wordpress version, add your current blog’s code
  1. Don’t add your private configs (wp-config.php)!
  2. Don’t add uploads folder
Atention with your private files!!
Update your wp-config
1. All your private configs must use getenv, this function will be responsible to fetch the values from env vars.

    <?php
    define('AUTH_KEY',         getenv('WP_AUTH_KEY'));
    define('SECURE_AUTH_KEY',  getenv('WP_SECURE_AUTH_KEY'));
    define('LOGGED_IN_KEY',    getenv('WP_LOGGED_IN_KEY'));
    define('NONCE_KEY',        getenv('WP_NONCE_KEY'));
    
    define('AUTH_SALT',        getenv('WP_AUTH_SALT'));
    define('SECURE_AUTH_SALT', getenv('WP_SECURE_AUTH_SALT'));
    define('LOGGED_IN_SALT',   getenv('WP_LOGGED_IN_SALT'));
    define('NONCE_SALT',       getenv('WP_NONCE_SALT'));
    
    define('S3_UPLOADS_BUCKET', getenv('AWS_S3_BUCKET'));
    define('S3_UPLOADS_KEY', getenv('AWS_S3_KEY'));
    define('S3_UPLOADS_SECRET', getenv('AWS_S3_SECRET'));
    define('S3_UPLOADS_REGION', getenv('AWS_S3_REGION'));

Create a composer.json file to define requirements and packages versions
1. Exemple composer.json

    {
      "require" : {
          "php": ">=7.0.0"
      },
      "require-dev": {
      }
    }

Execute composer update to generate the composer.lock file

Update your .htaccess file to redirect your uploads to your S3 bucket

Update the url (at the 5th line) on the .htaccess to match your S3 and Bucket

<IfModule mod_rewrite.c>
 RewriteEngine On
 RewriteBase /
 RewriteRule ^index\.php$ - [L]
 RewriteRule ^wp-content/uploads/(.*)$ https://s3-us-west-2.amazonaws.com/BUCKET/uploads/$1 [R=301,L]
 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteRule . /index.php [L]
 </IfModule>

Do a commit with all this files (don’t add your secrets/keys to your git)
Amazon setup
1. Create a RDS Database
2. Import your backup into it
3. Send your S3 files to S3
  1. Remember to import/change the permissions of your s3 files so guest users can access your uploaded files
Heroku setup
1. Add your environment vars on heroku with the correct values and names which you used on wp-config.php, remember to use the RDS ones for database.
2. Update your DNS for Heroku
3. Send your code to Heroku using the repository you have created
Access your website

If you’re updating your Wordpress, there are chances to something go wrong or to some plugin to stop working with the new Wordpress version, so don’t forget to check and update them.

Also, if you have any other question or need more information a specific test, let me know. I can try to help.

Matheus

Comment

Matheus Bratfisch Cogito ergo sum

Steps:

Steps: