Click here to Skip to main content
13,259,976 members (49,362 online)
Click here to Skip to main content
Add your own
alternative version

Stats

3.5K views
2 bookmarked
Posted 19 Jul 2017

How to integrate elasticsearch with ZF3

, 19 Jul 2017
Rate this:
Please Sign up or sign in to vote.
In the digital world, we are in Machine Leaning Phase. Where are need to everything on lighting speed. Data storing as we need, in our custom formant, and their availability, stability should be done on finger tips with low infrastructural cost.

Introduction

In the digital world, we are in Machine Leaning Phase. Where are need to everything on lighting speed. Data storing as we need, in our custom formant, and their availability, stability should be done on finger tips with low infrastructural cost. So I am proposing the solution open source solution with now cost.

We are going to us APACHE as web server, PHP an scripting language, Elasticsearch as NOSQL, non-structured database and Zend Framework 3 (ZF3) as development framework. So we are using LAMP (completely open source and no cost involve).

Background

I was looking for secure, robust, loose binded, agile, configurable, superfast data availability and the budged, hence chosen then Ubuntu OS, Apache web server, PHP web scripting complier and Elasticsearch as robust, non-structured database.

Business Value:

  • Cost effective
  • Reducing Development and Maintains cost
  • Improving the business process

Using the code

Before starting I assume that development environment is up and running, however ready then please find the below list to setup the environment.  

Now we are ready with environment, now lest setup the module in ZF3 for fetching the records / document from our non-relations database, before that please insert the data in Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_indexing_documents.html).

 

Step 1:  

Add the Elasticsearch library in the root “vendor” folder. You can get the Elasticsearch library from https://github.com/elastic/elasticsearch-php

 

Step2:

Open the “composer.json” file and add the below code in size “require” array

"require": {
    .....
    "elasticsearch/elasticsearch": "^5.0",
    .....
}

 

Step 3:

Creat log folder for to tracking application level error. So in the root directory we are creating the “data/log” folder. And giving the read, write and execute permission to “log” folder.

 

Steps 4:

Not let’s create the Elasticsearch connection configuration details, which is accessible across the all application module. So open the “config/autoload” folder and add the “local.php” file if not exists. Same file are got used to adding the local configuration details. Add below Elasticsearch connection configuration in same file such as…

return[
…
'elasticsearch' => [
        'connections' => [
             'justbuylive' => [
                 'index'  => "es_local",
                 'host'   => 'localhost',
                 'port'   => '9200',
                 'scheme' => 'http'
             ]
         ],
        'type'=>[
            'ES_CLIENT_MASTER'=>'client_master'
        ]
    ],
..
]

 

Step 5:

Now it’s time to creating “modules” with routing and configuration so first thing first. Now creating our custom module with name “Api”. For same our folder structure should be look like

[project name]
--config
----autoload
------local.php
----module.config.php
--data
----log
--module
----Api
------config
--------module.config.php
------src
--------Controller
----------Factory
------------RetialerdataserviceControllerFactory.php
----------Plugin
------------ DefaultPlugin.php
------------Factory
------------RetialerdataserviceController.php
--------Entity
--------Listener
--------Repository
------------RetailerdataserviceRepository.php
--------Service
----------Factory
------------RetialerdataserviceManagerFactory.php
----------RetailerdataserviceManager.php
--------Module.php
--vendor
----elasticsearch
------elasticsearch

 

Step 6:

Creating “module/Api/config/module.config.php” Api module configuration and routing file. Contains below code…

<!--?php
/**
 * @link      http://github.com/zendframework/ZendSkeletonApplication for the canonical source repository
 * @copyright Copyright (c) 2005-2016 Zend Technologies USA Inc. (http://www.zend.com)
 * @license   http://framework.zend.com/license/new-bsd New BSD License
 */

namespace Api;

use Doctrine\ORM\Mapping\Driver\AnnotationDriver;
use Zend\Router\Http\Literal;
use Zend\Router\Http\Segment;

return [
    'doctrine' =----> [
        'driver' => [
            __NAMESPACE__ . '_driver' => [
                'class' => AnnotationDriver::class,
                'cache' => 'array',
                'paths' => [__DIR__ . '/../src/Entity']
            ],
            'orm_default' => [
                'drivers' => [
                    __NAMESPACE__ . '\Entity' => __NAMESPACE__ . '_driver'
                ],
            ],
            'orm_accesslog' => [
                'drivers' => [
                    __NAMESPACE__ . '\Entity' => __NAMESPACE__ . '_driver'
                ],
            ], 
        ]
    ],
    'router' => [
        'routes' => [
            'retialerdataservice' => [
                'type' => Literal::class,
                'options' => [
                    'route' => '/getretailer',
                    'defaults' => [
                        '__NAMESPACE__' => 'Api\Controller',
                        'controller' => Controller\RetialerdataserviceController::class,
                        'action'     => 'index',
                    ],
                ],
            ],
            '/' => [
                'type' => Literal::class,
                'options' => [
                    'route' => '/',
                    'defaults' => [
                        '__NAMESPACE__' => 'Api\Controller',
                        'controller' => Controller\IndexController::class,
                        'action'     => 'index',
                    ],
                ],
            ],
  ],
    ],
     'controllers' => [
        'factories' => [
            Controller\RetialerdataserviceController::class => Controller\Factory\RetialerdataserviceControllerFactory::class,
        ],
    ],
    'controller_plugins' => [
        'invokables' => [
            'DefaultPlugin' => 'Api\Controller\Plugin\DefaultPlugin',
        ]
    ],
    'service_manager' => [
        'factories' => [
Service\RetialerdataserviceManager::class => Service\Factory\RetialerdataserviceManagerFactory::class,
            'ElasticSearch' => Service\Factory\ElasticClientFactory::class,
        ]
    ],
    'view_manager' => [
        'strategies' => [
            'ViewJsonStrategy',
        ],
    ],
];

 

Step 7:

Now it’s time to create the controller, which is responsible to perform / process  the requested. “module/Api/src/Controller/RetialerdataserviceController.php”. controller contains below line of code…

<!--?php
/**************************************************************************
 * Purpose   : Manging the retailer data request and response 
 * Created By: Jaiswar Vipin Kumar R.
 **************************************************************************/

namespace Api\Controller;
use Zend\Mvc\Controller\AbstractActionController;
use Zend\View\Model\JsonModel;

class RetialerdataserviceController extends AbstractActionController{
    /* Varaibe delclaration */
    private $_objRetailerDataServiceObject;

    /***************************************************************************
     * Purpose      : Default method get executed.
     * Inputs       : $pObjRetailerDataServiceManager :: Retailer Data servier 
     *                                                   Manager object refrence
     * Return       : None.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function __construct($pObjRetailerDataServiceManager){
        /* Assign manager refrence */
        $this--->_objRetailerDataServiceObject = $pObjRetailerDataServiceManager;
    }

    /***************************************************************************
     * Purpose      : First Operation action need to be execute to process the 
     *                request.
     * Inputs       : None.
     * Return       : None.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function indexAction(){
        /* Getting the consumer service request */
        $strRequestContentArr   = json_decode($this->getRequest()->getContent(), true);
        $strResponseDataArr     = array("status" => 0, "message" => "Invalid Requests.");
        
        if(!empty($strRequestContentArr)){
            $strResponseDataArr = $this->_objRetailerDataServiceObject->getRetailerDetails($strRequestContentArr);
        }else{
            $strResponseDataArr = array("status" => 0, "message" => "Invalid Requests.");
        }
        /* Setting data log */
        $this->setLog($strResponseDataArr);
        
        /* return the response is JSON Model Format */
        return  new JsonModel($strResponseDataArr);
    }

    /***************************************************************************
     * Purpose      : Setting log.
     * Inputs       : $pDataSet :: Data Set.
     * Return       : None.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function setLog($pDataSet){
        /* Creating Today's log file name */
        $strFileName = date('Y-m-d');

        /* Setting the data log folder relative path */
        $strDataLogPath = 'data/log';
        
        /* Checking is data log exists */
        if (!file_exists($strDataLogPath) && !is_dir($strDataLogPath)) {
            /* if not exists then create with User - Group - Other RWX permission */
            mkdir($strDataLogPath, 0777, true);      
        }
        
        /* Setting the file name path */
        $strFileName = $strDataLogPath."/".$strFileName.".txt";
        
        /* Creating log data set */
        $strLogData = date('Y-m-d H:i:s')."\n----".json_encode($pDataSet)."\n";

        /* writing in the log file */
        file_put_contents($strFileName, $strLogData, FILE_APPEND | LOCK_EX);
        
        /* removed used variables */
        unset($strLogData, $strFileName, $strDataLogPath);
    }
}

 

Step 8:

Now create the factory class of the controller which helps to make “Service Manager” avabible in the controller and process the business logic and return the desired output by help of _invoking() auto execute method. Navigating to “module/Api/src/Controller/Factory/RetialerdataserviceControllerFactory.php” factory  controller contains below line of code…

<?php 
namespace Api\Controller\Factory;

use Interop\Container\ContainerInterface;
use Zend\ServiceManager\Factory\FactoryInterface;
use Api\Controller\RetialerdataserviceController;
use Api\Service\RetialerdataserviceManager;

class RetialerdataserviceControllerFactory implements FactoryInterface{
    public function __invoke(ContainerInterface $container,$requestedName,array $options = null){
        $retialerdataserviceManager = $container->get(RetialerdataserviceManager::class);			
        // Instantiate the controller and inject dependencies
        return new RetialerdataserviceController($retialerdataserviceManager);
    }
}

?>

 

Step 9:

Now creating the global method for making operation life easy using oops. Now creating the plugging for same with name “module/Api/src/Controller/Plugin/ DefaultPlugin.php” global   resource, available across the Api module, contains below line of code…

<?php
namespace Api\Controller\Plugin; 

use Zend\Mvc\Controller\Plugin\AbstractPlugin;
use Zend\View\Model\JsonModel;
use Zend\Config\Factory;

/* 
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */

class DefaultPlugin extends AbstractPlugin{
    /*****************************************************************************************/
    /* Purpose  : Creating the result set as array response.
     * Inputs   : $pObjResultSet :: Result Set,
     *          : $pIntResultSetType :: 0: MYSQL, 1: Elasticsearch, 2: Postgrays
     * Returns  : Result set array.
     * CreatedBy: Jaiswar VIpin Kumar R.
    /*****************************************************************************************/
    public function getResultset($pObjResultSet = null, $pIntResultSetType = 0){
        /* Variable initialization */
        $strReturnArr           = array();
        $blnRecordExists        = false;
        $objResultSetContainer  = null;
        
        /* Based on the data source type checkig the result set */
        switch($pIntResultSetType){
            /* MYSQL */
            case 0:
                /* if result set not empty then do needful */
                if(!empty($pObjResultSet)){
                    /* Value overridding */
                   $blnRecordExists         = true;
                   $objResultSetContainer   = $pObjResultSet;
                }
                break;
            /* Elasticsearch */
            case 1:
                /* if result set not empty then do needful */
                if(isset($pObjResultSet['hits']['total']) && ($pObjResultSet['hits']['total'] > 0)){
                    /* Value overridding */
                   $blnRecordExists         = true; 
                   $objResultSetContainer   = $pObjResultSet['hits']['hits'];
                }
                break;
            /* Postgray */
            case 2:
                break;
        }
        
        /* if record exists then do needful */
        if($blnRecordExists){
            /* iterating the loop */
            foreach($objResultSetContainer as $objResultSetContainerKey => $objResultSetContainerValue){
                /* Based on the data source type checkig the result set */
                switch($pIntResultSetType){
                    /* MYSQL */
                    case 0:
                        /* Setting the value in array format */
                        $strReturnArr[] = (array)$objResultSetContainerValue;
                        break;
                    /* Elasticsearch */
                    case 1:
                        /* Setting the value in array format */
                        $strReturnArr[] = $objResultSetContainerValue['_source'];
                        break;
                    /* Postgray */
                    case 2:
                        break;
                }
            }
        }
        
        /* removed used variabled */
        unset($blnRecordExists, $objResultSetContainer);
        
        /* return the result set */
        return $strReturnArr;
    }
}

 

Step 10:

Now creating the repository class for processing the request, same repository class get invoke from service manager class. This class contains the business processing. Now creating the repository for processing the “module/Api/src/Repository/RetailerdataserviceRepository.php”. Class contains below line of code…

<?php
/**************************************************************************
 * Purpose   : Manging the retailer DML / ORM operation 
 * Created By: Jaiswar Vipin Kumar R.
 **************************************************************************/

namespace Api\Repository;

use Api\Entity\ClientMaster;
use Api\Controller\Plugin\DefaultPlugin;
use Doctrine\ORM\EntityRepository;
use Elasticsearch\ClientBuilder;
use Elasticsearch\Transport;
use Zend\View\Model\JsonModel;

class RetailerdataserviceRepository extends EntityRepository{
    
    private $_strEntityArr  = array();
    
    
    /***************************************************************************
     * Purpose      : Get the retailer details.
     * Inputs       : None.
     * Return       : Retailer Details.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function getRetailerDetailsFromElastica($pStrFilterArr){
        /* Variable initialization */
        $strReturnArr   = array();
        
        try{
            /* Getting the local file refrence object */
            $objElasticConfiguration    = DefaultPlugin::getLocalCofigurationByKey('elasticsearch');
            
            /* if configuration is not set then do needful */
            if(empty($objElasticConfiguration)){
                /* return error message */
                return array("status" => 0, "message" => "Elasticsearch Configuration is not set");
                /* Stop the execution */
                exit;
            }
            
            /* Getting ES connection objects */
            $objESSettings = $objElasticConfiguration->connections->justbuylive;
            /* Creating ES hosts array */
            $strHostArray[] = $objESSettings->host.':'.$objESSettings->port;
            
            /* Creating Elasticssearch Objects */
            $esClientObj = ClientBuilder::create()->setHosts($strHostArray)->setRetries(2)->build();
            /* Get ES filter */
            $strSearchFilterArr    = $this->_getFilterCritriaElasticsearch($pStrFilterArr);
            
            /* Filter Array */
            $strFilterArr = [
                                'index'  => $objElasticConfiguration->connections->justbuylive->index,
                                'type'   => $objElasticConfiguration->type->ES_CLIENT_MASTER,
                                'body' => []
                            ];
            $strFilterArr['body']['query']['bool']['filter']    = $strSearchFilterArr;
            
            /* removed used variables */
            unset($objElasticConfiguration, $objESSettings, $strHostArray, $strSearchFilterArr);
            
            /* Fetching resule set and creating the return result set */
            $strReturnArr = DefaultPlugin::getResultset($esClientObj->search($strFilterArr), 1);
            
            /* removed used variables */
            unset($esClientObj);
            
        } catch (Exception $ex) {
             /* return error message */
            return array("status" => 0, "message" => $ex->getMessage());
            /* Stop the execution */
            exit;
        }
        
        /* return resultset */
        return $strReturnArr;
    }
}

 

Step 11:

Now creating service manager for giving the service to the API module as worker Now creating the repository for processing the “module/Api/src/Service/RetailerdataserviceManager.php”. Class contains below line of code…

<?php
/**************************************************************************
 * Purpose   : Retailer Data Service Manager to process the configuration
 *             DQL and user request and return the appropriate response.
 * Created By: Jaiswar Vipin Kumar R.
 **************************************************************************/

namespace Api\Service;

use Api\Entity\ClientMaster;
use Exception;
use Zend\Config\Factory;

class RetailerdataserviceManager{
    /**
     * Doctrine entity manager.
     * @var Doctrine\ORM\EntityManager
     */
    /* varaible initialization */
    private $_entityManager;

    /***************************************************************************
     * Purpose      : Default method get executed in the manager
     * Inputs       : $pObjRetailerDataServiceEntityManager :: Retailer Data servier 
     *                Manager object refrence
     * Return       : None.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function __construct($pObjRetailerDataServiceEntityManager) {
        $this->_entityManager = $pObjRetailerDataServiceEntityManager;
    }

    /***************************************************************************
     * Purpose      : Get the requested retailer details.
     * Inputs       : $pStrRequestArr :: Customer request array.
     * Return       : Retailer data set is any.
     * Created By   : Jaiswar Vipin Kumar R.
    ****************************************************************************/
    public function getRetailerDetails($pStrRequestArr){
       /* Return the Retailer details */
            return $this->_entityManager->getRepository(ClientMaster::class)->getRetailerDetailsFromElastica($pStrRequestArr);
    }
}

 

Step 12:

Now creating service manager factory class which actually work an worker and helps to service manger to complete the module. Now creating the factory for processing the “module/Api/src/Service/Factory/ RetialerdataserviceManagerFactory.php”. Class contains below line of code…

<?php
/***************************************************************************************************
 * Purpose   : This is the factory class for ConfigManager service. The purpose of the factory
 *             is to instantiate the service and pass it dependencies (inject dependencies)
 * Created By: Jaiswar Vipin Kumar R.
 ***************************************************************************************************/
namespace Api\Service\Factory;

use Interop\Container\ContainerInterface;
use Api\Service\RetailerdataserviceManager;

class RetialerdataserviceManagerFactory{
    /***************************************************************************************************
     * Purpose  : This method creates the ConfigManager service and returns its instance. 
     * Inputs   : $container :: Contianer Interface instant object,
     *          : $requestedName :: Request handller Name
     *          : $options  :: Configuration Variables
     * Returns  : Service configuration manager object
     ***************************************************************************************************/
    public function __invoke(ContainerInterface $container, $requestedName, array $options = null){   
        /* Getting the ORM connection configuration value */
        $entityManager = $container->get('doctrine.entitymanager.orm_default');
        
        /* Return the retailer manager insetance */
        return new RetailerdataserviceManager($entityManager);
    }
}

 

Note: I havent' added the entity class "ClientMaster", so you can create your own ORM entity class and excust the same.

Happy Coding !!!

Points of Interest

  1. Business Value
  2. Operational Value
  3. Cost Saving
  4. Reusable

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Jaiswar Vipin Kumar R.
Program Manager
India India
This member doesn't quite have enough reputation to be able to display their biography and homepage.
Group type: Organisation (No members)



You may also be interested in...

Pro

Comments and Discussions

 
-- There are no messages in this forum --
Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.171114.1 | Last Updated 20 Jul 2017
Article Copyright 2017 by Jaiswar Vipin Kumar R.
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid