One Openemm - 4 computers. Is this possible?

Use this forum for all questions related to the source code of OpenEMM

Moderator: moderator

One Openemm - 4 computers. Is this possible?

Postby sfantul » Thu Sep 28, 2006 2:40 pm

I want to broke openemm in modules that stay on different computers.
Mashine 1: the database
Mashine 2: python ,c scripts and sendmail
Mashine 3: openemm interface
Mashine 4: openemm redirection server

Is this possible?
What i want to know to be more exact is how this 4 entities interact.
What i know:
The database is easy to separate from others .
-for this we have to properlly modify emm.properties ,agn.py and Mailgun.ini (By the way who uses this file beside webapps? and if nobody else uses it why cant be used just "emm.properties")

Both webapps write some log files to disk in the openemm file system (but i dont think this is a big problem they could write them anywhere. no other script use them).

Console app write some xml to disk when sending mails that the other scripts fetch. This is the first big problem.

Are there any other problems?
The redirection app for example have any contact with the filesystem ? (beside log files of course)
sfantul
 
Posts: 40
Joined: Tue Aug 15, 2006 9:34 am

Postby sfantul » Thu Nov 02, 2006 1:24 pm

I'll answer myself .
Openem can be relatively easy split on 3 mashines .
1) Openemm user, filesystem , python and c script, sendmail and openemm interface
2)database
3)Redirection server.
This is important because the redirection server could be put in a location where a good uptime is ensured and also the database.
The interface doesnt need to be online all the time.
sfantul
 
Posts: 40
Joined: Tue Aug 15, 2006 9:34 am

Postby marton » Sun Feb 18, 2007 4:30 pm

hi,

does it work? do you run openemm on three machines?

thanks in advance, marton
marton
 
Posts: 36
Joined: Wed Aug 09, 2006 2:51 pm

Postby sfantul » Mon Feb 19, 2007 9:47 am

yes, I do , i have :
- the database on a mashine
- the interface , python scripts, log files , etc on an other mashine
- redirection (public) server on another mashine
sfantul
 
Posts: 40
Joined: Tue Aug 15, 2006 9:34 am

distributed openemm

Postby maschoff » Wed Feb 21, 2007 10:23 am

sfantul, I am curious: How many mails per hour can you send with this configuration?
OpenEMM Maintainer
maschoff
Site Admin
 
Posts: 2144
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany

Postby sfantul » Wed Feb 21, 2007 11:45 am

Currently i have a mailing list of 600k subscribers and i limit the mailing to 50k per hour (but it can do better).I did this limitation because of the notifications that i receive from some domains that my mailing rate is too high.
But the mailing rate has nothing to do to with this configuration , its all about configuring sendmail (when i started i had a rate of 10k per hour max). I tried a lot of things to make the sending rate higher.

First big breakthrough was to split the queue folder in 10 folders with persistent runners on every queue
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE0
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE1
...
$sm -qp1s -NFAILURE -OQueueDirectory=$BASE/var/spool/QUEUE9
changing pickdist.py and mailer.sh accordingly and sorting the queue after the host. here are my defines in sendmail.mc:
define(`confQUEUE_LA', `50')dnl
define(`confREFUSE_LA', `51')dnl
define(`confCON_EXPENSIVE',`true')dnl
define(`confTO_ICONNECT', `5s')dnl
define(`confQUEUE_SORT_ORDER', `Host')dnl
define(`confMIN_QUEUE_AGE', `3h')dnl

This was a big improvement, but finally i'm using an extra postfix mailer as a smarthost which give me the results i was talking about.
The disadvantages of this last solution are that u need to also configure an postfix mailer and that the first type of bounce management does not work properlly.
sfantul
 
Posts: 40
Joined: Tue Aug 15, 2006 9:34 am

Postby joedj » Tue Jun 10, 2008 5:07 am

Hi sfantul,

We are interested in doing something similar. Can you tell us what kind of changes you made to pickdist.py to use the multiple queues, or provide a copy? I know this was a while ago now... :)
joedj
 
Posts: 2
Joined: Tue Apr 22, 2008 2:46 am

Postby dcunniff » Wed Oct 29, 2008 3:43 pm

Hi guys,

I'm looking for the same information if it could be provided. I too understand this was a long time ago.
dcunniff
 
Posts: 4
Joined: Tue Oct 14, 2008 10:11 pm

Re: One Openemm - 4 computers. Is this possible?

Postby tpalmieri » Tue Dec 13, 2011 1:24 pm

I am trying to host openemm pages, etc on one server and the database on another.

I edited both cms.properties and emm.properties to point to the remote database server, but openemm times out when it tries to connect.

cms.properties:
#####################################################
# Database settings
#####################################################

cmsdb.driverClassName=com.mysql.jdbc.Driver
cmsdb.url=jdbc:mysql://internal.c3psypivq9bi.us-east-1.rds.amazonaws.com:3306/openemm_cms?useUnicode=true&characterEncoding=UTF8&jdbcCompliantTruncation=false
cmsdb.dialect=org.hibernate.dialect.MySQLDialect
cmsdb.username=agnitas
cmsdb.password=openemm
cmsdb.maxCount=30
cmsdb.maxWait=10000
cmsdb.validationQuery=SELECT 1 FROM DUAL

#####################################################
#Remote webservices settings
#####################################################
cms.ccr.url=http://openemm.simpractice.com:8080

emm.properties:
# Property file with OpenEMM settings - targeted at system administrators

###############################################################################
# Database Connection Settings (dataAccessContxt.xml, etc.)
###############################################################################
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://internal.c3psypivq9bi.us-east-1.rds.amazonaws.com:3306/openemm?useUnicode=yes&characterEncoding=UTF-8&useOldAliasMetadataBehavior=true
jdbc.dialect=org.hibernate.dialect.MySQLDialect
jdbc.username=agnitas
jdbc.password=openemm
jdbc.maxCount=30
jdbc.maxWait=10000
jdbc.validationQuery=SELECT 1 FROM DUAL

###############################################################################
# System Default Values
###############################################################################
system.url=http://openemm.simpractice.com:8080
system.logdir=/home/openemm/var/log
system.upload=/tmp
system.updateserver=http://www.openemm.org/
system.mail.host=localhost

###############################################################################
# Feature Default Values
###############################################################################
password.expire.days=-1
fckpath=fckeditor-2.6.4.1
ecs.server.url=http://openemm.simpractice.com:8080

Any help anyone can provide would be greatly appreciared!

Thanks!
tpalmieri
 
Posts: 11
Joined: Tue Dec 13, 2011 1:16 pm

Re: One Openemm - 4 computers. Is this possible?

Postby tpalmieri » Tue Dec 13, 2011 5:07 pm

Disregard my last post, I got it all working:

Web Server runs the pages
Database Server holds all the data

The web server is running on Amazon EC2, and the Database server is running on Amazon's RDS service.

I'm uploading about 75,000 records now and its going pretty fast (although I have no frame of reference!)

LOVE OpenEMM!
tpalmieri
 
Posts: 11
Joined: Tue Dec 13, 2011 1:16 pm

Re: One Openemm - 4 computers. Is this possible?

Postby maschoff » Tue Dec 13, 2011 6:11 pm

Perhaps is would be interesting for other OpenEMM users to share your settings (in case you want to disclose your knowhow).
OpenEMM Maintainer
maschoff
Site Admin
 
Posts: 2144
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany

Re: One Openemm - 4 computers. Is this possible?

Postby unhuman » Tue Mar 13, 2012 9:52 pm

Pretty sure we're going to explore this as well. I currently envision:
1. OpenEMM itself (UI + Mail sending)
2. Database
3. Bounce Detection
4. Click detection
5. We'll either use Amazon SES or stand up our own STMP server(s). I might add support for multiple configurations so this data can be sent round-robin style.

To note, I'm not using OpenEMMs send logic (with the subprocesses). I'm using my own mail building code (for SES and/or SMTP) and will be writing a client to detect bounces.

We may combine 3 & 4 since they should be lightweight processes, however, we'll be sending out mail in the hundreds of thousands / millions, so, we might want more girth there. We could easily use a VIP to allow us to map multiples of those.

-H
unhuman
 
Posts: 46
Joined: Mon Jan 23, 2012 10:29 pm

Re: One Openemm - 4 computers. Is this possible?

Postby Infomess » Mon Oct 08, 2012 2:17 pm

This is a great post! I was wondering if you could help me.

I have installed OpenEMM 2012 on an EC2 instance and it looks like it's working fine. Now created a RDS instance which I intend to use for the DB. I have
updated the emm and css.properties file as tpalmieri has indicated. OpenEMM works fine, I can logon to CMS but when starting openmm.

It says "Failed to setup database interface".

I have also updated the address from localhost to the rds endpoint on agn.py but still no luck.

Running with the database on the same instance everything is working fine.

Any idea how to debug this?

Many many thanks!
Infomess
 
Posts: 3
Joined: Mon Oct 08, 2012 2:05 pm


Return to Development

Who is online

Users browsing this forum: No registered users and 3 guests