Loading ...
Sorry, an error occurred while loading the content.
 

canonical and milter

Expand Messages
  • Christian Rößner
    Hi, I have this in my logs: Apr 27 03:42:59 mx postfix/smtpd[16599]: connect from outmail038.prn2.facebook.com[66.220.144.165]:61593 Apr 27 03:42:59 mx
    Message 1 of 6 , Apr 27, 2014
      Hi,

      I have this in my logs:

      Apr 27 03:42:59 mx postfix/smtpd[16599]: connect from outmail038.prn2.facebook.com[66.220.144.165]:61593
      Apr 27 03:42:59 mx postfix/smtpd[16599]: Anonymous TLS connection established from outmail038.prn2.facebook.com[66.220.144.165]:61593: TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)

      Apr 27 03:43:00 mx smf-spf[19950]: SPF pass: ip=66.220.144.165, fqdn=outmail038.prn2.facebook.com, helo=mx-out.facebook.com, from=<apps+kr4yybbknqwr@...>

      Apr 27 03:43:00 mx postfix/smtpd[16599]: 3gGX1w5JWhzyQD: client=outmail038.prn2.facebook.com[66.220.144.165]:61593

      Apr 27 03:43:00 mx postsrsd[16727]: srs_forward: <apps+kr4yybbknqwr@...> rewritten as <SRS0+svZH=Z3=facebookappmail.com=apps+kr4yybbknqwr@...>

      Apr 27 03:43:00 mx postfix/cleanup[16726]: 3gGX1w5JWhzyQD: message-id=<41c610fe94a6007909593173eaa5c193@...>

      Apr 27 03:43:02 mx amavis[24491]: (24491) Passed CLEAN {AcceptedInbound}, AM.PDP-SOCK [66.220.144.165] [66.220.144.165] <apps+kr4yybbknqwr@...> -> <XXXX@XXXXX>, Queue-ID: 3gGX1w5JWhzyQD, Message-ID: <41c610fe94a6007909593173eaa5c193@...>, mail_id: 4ujS73lHgOJC, Hits: 1.402, size: 5209, Tests: [BAYES_00=-0.2,DCC_CHECK=1.1,FROM_LOCAL_NOVOWEL=0.5,HTML_IMAGE_ONLY_32=0.001,HTML_MESSAGE=0.001,RCVD_IN_DNSWL_NONE=-0.0001,SPF_PASS=-0.001,UNPARSEABLE_RELAY=0.001], 1301 ms
      Apr 27 03:43:02 mx amavisd-milter[3816]: 3gGX1w5JWhzyQD: log_id=24491
      Apr 27 03:43:02 mx amavisd-milter[3816]: 3gGX1w5JWhzyQD: return_value=continue

      Apr 27 03:43:02 mx opendkim[10497]: 3gGX1w5JWhzyQD: outmail038.prn2.facebook.com [66.220.144.165] not internal
      Apr 27 03:43:02 mx opendkim[10497]: 3gGX1w5JWhzyQD: not authenticated
      Apr 27 03:43:02 mx opendkim[10497]: 3gGX1w5JWhzyQD: not POP authenticated

      Apr 27 03:43:02 mx opendmarc[4591]: 3gGX1w5JWhzyQD: facebookappmail.com fail

      Apr 27 03:43:02 mx postfix/cleanup[16726]: 3gGX1w5JWhzyQD: milter-reject: END-OF-MESSAGE from outmail038.prn2.facebook.com[66.220.144.165]: 5.7.1 rejected by DMARC policy for facebookappmail.com; from=<SRS0+svZH=Z3=facebookappmail.com=apps+kr4yybbknqwr@...> to=<ep.walter@...> proto=ESMTP helo=<mx-out.facebook.com>

      Apr 27 03:43:07 mx postfix/smtpd[16599]: disconnect from outmail038.prn2.facebook.com[66.220.144.165]:61593

      There are four milters:

      1. smf-spf
      2. OpenDKIM
      3. OpenDMARC
      4. amavisd-new

      I tried to use postsrsd to get forwarding done. The mailserver is a provider mailserver. Multi domains. Some accounts end on this server, some mail addresses are forwarded (unfortunately).

      I found postsrsd and thought that might solve my problem.

      What I do not understand:

      I thought mail would arrive on smtpd where all the milters are called and afterwards the mail would be handed over to cleanup, which does canonical stuff. But it seems I am wrong :)

      First the configuration parts that describe my problem:


      In main.cf:
      sender_canonical_maps = tcp:[::1]:10001
      sender_canonical_classes = envelope_sender
      recipient_canonical_maps = tcp:[::1]:10002
      recipient_canonical_classes = envelope_recipient
      #
      relay_transport = lmtp:[::1]:24
      relay_domains = ${mapidx}/relay_domains
      relay_recipient_maps = ${mapidx}/relay_recipient_maps
      virtual_alias_maps = ${mapidx}/aliases, ${mapidx}/virtual


      In master.cf:
      smtpd pass - - - - - smtpd
      -o smtpd_milters=inet:[::1]:30065,inet:[::1]:10024,inet:[::1]:8891,inet:[::1]:8893
      -o cleanup_service_name=cleanup2
      -o smtpd_delay_reject=no
      cleanup2 unix n - - - 0 cleanup
      -o header_checks=pcre:${map}/header_checks.pcre,regexp:${map}/add_header.regexp
      -o body_checks=pcre:${map}/body_checks.pcre


      It would be nice, if I knew how to tell Postfix that it does canonicalization _after_ smtpd/milter. But it must do all the virtual_alias stuff. So receive_override_options=no_address_mapping does not work.

      I am stuck on this :) Maybe you like to help me.

      Thanks in advance

      -Christian Rößner

      Here is the complete config (if I forgot some important detail) postsrsd is disabled currently, as I need a fix first:


      postfinger - postfix configuration on Sun Apr 27 19:55:29 CEST 2014
      version: 1.30

      Warning: postfinger output may show private configuration information,
      such as ip addresses and/or domain names which you do not want to show
      to the public. If this is the case it is your responsibility to modify
      the output to hide this private information. [Remove this warning with
      the --nowarn option.]

      --System Parameters--
      mail_version = 2.11.0
      hostname = mx
      uname = Linux mx 3.13.6-hardened-r3 #1 SMP Tue Apr 8 16:11:11 CEST 2014 x86_64 QEMU Virtual CPU version 1.0 GenuineIntel GNU/Linux

      --Packaging information--

      --main.cf non-default parameters--
      alias_database = ${default_database_type}:/etc/aliases, ${default_database_type}:/etc/mail/aliases
      alias_maps = ${default_database_type}:/etc/aliases, ${default_database_type}:/etc/mail/aliases
      anvil_rate_time_unit = 30s
      biff = no
      bounce_queue_lifetime = 1d
      bounce_template_file = ${config_directory}/bounce.de-DE.cf
      broken_sasl_auth_clients = yes
      debugger_command = PATH=/bin:/usr/bin:/usr/local/bin; (strace -p ${process_id} 2>&1 | logger -p mail.info) & sleep 5
      default_database_type = cdb
      delay_warning_time = 2h
      disable_vrfy_command = yes
      enable_long_queue_ids = yes
      header_checks = regexp:${map}/add_header.regexp
      inet_interfaces = ${mx_deltaweb_de}
      inet_protocols = ipv4, ipv6
      localhost_smtpd_recipient_restrictions = check_recipient_access pcre:${map}/roleaccount.pcre, permit_mynetworks, reject
      mailbox_size_limit = 0
      mailout_deltaweb_de = 193.239.107.53
      map = ${config_directory}/maps
      mapidx = ${default_database_type}:${map}
      maximal_queue_lifetime = 1d
      message_size_limit = 31457280
      milter_connect_macros = j {daemon_name} {client_ptr} v
      milter_default_action = accept
      milter_mail_macros = i {auth_type} {auth_authen} {auth_author} {mail_addr} {mail_host} {mail_mailer} {client_name}
      minimal_backoff_time = 5m
      mx_deltaweb_de = 193.239.107.52
      mydestination = ${myhostname}, localhost
      mydomain = deltaweb.de
      myhostname = mx.${mydomain}
      mynetworks = ${mailout_deltaweb_de} ${relay_deltaweb_de}
      owner_request_special = no
      parent_domain_matches_subdomains =
      postscreen_access_list = cidr:${map}/postscreen_blacklist.cidr, cidr:${map}/postscreen_whitelist.cidr
      postscreen_blacklist_action = enforce
      postscreen_cache_map = memcache:${map}/postscreen_cache.cf
      postscreen_dnsbl_action = enforce
      postscreen_dnsbl_sites = zen.spamhaus.org*3, ix.dnsbl.manitu.net*3, dsn.rfc-ignorant.de*2, bl.spamcop.net*1, b.barracudacentral.org*1, swl.spamhaus.org*-2
      postscreen_dnsbl_threshold = 3
      postscreen_dnsbl_whitelist_threshold = -1
      postscreen_greet_action = enforce
      proxy_write_maps = proxy:btree:${data_directory}/postscreen_cache
      queue_minfree = 47185920
      recipient_delimiter = +
      relay_deltaweb_de = 193.239.107.55
      relay_domains = ${mapidx}/relay_domains
      relay_recipient_maps = ${mapidx}/relay_recipient_maps
      relay_transport = lmtp:[::1]:24
      roleaccount = check_sender_access ${mapidx}/sender_access, check_client_access pcre:${map}/client_access.pcre, check_client_access cidr:${map}/client_access.cidr, check_helo_access pcre:${map}/helo_access.pcre, check_helo_access ${mapidx}/check_helo, check_recipient_access pcre:${map}/roleaccount.pcre, permit
      show_user_unknown_table_name = no
      smtp_bind_address = ${mx_deltaweb_de}
      smtpd_authorized_verp_clients = ${mynetworks}
      smtpd_banner = ${myhostname} ESMTP
      smtpd_client_connection_rate_limit = 8
      smtpd_client_event_limit_exceptions = ${mynetworks}, 193.239.104.0/22
      smtpd_client_message_rate_limit = 20
      smtpd_client_new_tls_session_rate_limit = 5
      smtpd_client_port_logging = yes
      smtpd_command_filter = pcre:${map}/command_filter.pcre
      smtpd_data_restrictions = reject_multi_recipient_bounce
      smtpd_etrn_restrictions = reject
      smtpd_helo_required = yes
      smtpd_milters = inet:[::1]:10024
      smtp_dns_support_level = dnssec
      smtpd_policy_service_timeout = 5m
      smtpd_proxy_timeout = 300s
      smtpd_recipient_restrictions = reject_non_fqdn_recipient, reject_non_fqdn_sender, reject_unknown_recipient_domain, reject_unknown_sender_domain, reject_unlisted_recipient, permit_mynetworks, reject_unauth_destination, check_recipient_access pcre:${map}/roleaccount_exceptions.pcre, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, check_sender_access ${mapidx}/sender_access, check_client_access pcre:${map}/client_access.pcre, check_client_access cidr:${map}/client_access.cidr, check_client_access cidr:${map}/deltaweb_fax.cidr, check_helo_access pcre:${map}/helo_access.pcre, check_helo_access ${mapidx}/check_helo, reject_unknown_reverse_client_hostname, reject_unverified_recipient, check_policy_service inet:[::1]:12340
      smtpd_reject_footer = For assistance, see http://www.roessner-network-solutions.com/mail.html. Please provide the following information in your problem report: time (${localtime}), client (${client_address}:${client_port}) and server (${server_name}).
      smtpd_restriction_classes = roleaccount
      smtpd_sasl_authenticated_header = yes
      smtpd_sasl_path = private/auth
      smtpd_sasl_security_options = noanonymous, noplaintext
      smtpd_sasl_tls_security_options = noanonymous
      smtpd_sasl_type = dovecot
      smtpd_submission_banner = ${myhostname} ESMTP Submission
      smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
      smtpd_tls_cert_file = /etc/ssl/certs/mx_deltaweb_de.crt
      smtpd_tls_dh1024_param_file = ${config_directory}/dh_2048.pem
      smtpd_tls_dh512_param_file = ${config_directory}/dh_512.pem
      smtpd_tls_exclude_ciphers = aNULL, MD5, DES, RC4
      smtpd_tls_key_file = /etc/ssl/private/mx_deltaweb_de.key
      smtpd_tls_loglevel = 1
      smtpd_tls_mandatory_ciphers = high
      smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
      smtpd_tls_received_header = yes
      smtpd_tls_security_level = may
      smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_session_cache
      smtpd_use_tls = yes
      smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
      smtp_tls_exclude_ciphers = aNULL, MD5, DES, RC4
      smtp_tls_mandatory_ciphers = high
      smtp_tls_mandatory_exclude_ciphers = aNULL, MD5, DES, RC4
      smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
      smtp_tls_note_starttls_offer = yes
      smtp_tls_protocols = !SSLv2, !SSLv3
      smtp_tls_session_cache_database = btree:${data_directory}/smtp_session_cache
      smtp_use_tls = yes
      strict_rfc821_envelopes = yes
      submission_smtpd_relay_restrictions = check_sasl_access ${mapidx}/sasl_access, check_sender_access ${mapidx}/sender_access, reject_non_fqdn_recipient, permit_sasl_authenticated, reject_unauthenticated_sender_login_mismatch, reject
      tls_preempt_cipherlist = yes
      tls_ssl_options = no_ticket, no_compression
      transport_maps = ${mapidx}/transport
      unknown_address_reject_code = 550
      unknown_hostname_reject_code = 550
      unverified_recipient_reject_code = 550
      unverified_recipient_reject_reason = Address lookup failed
      virtual_alias_maps = ${mapidx}/aliases, ${mapidx}/virtual

      --master.cf--
      smtp inet n - - - 1 postscreen
      smtpd pass - - - - - smtpd
      -o smtpd_milters=inet:[::1]:30065,inet:[::1]:10024,inet:[::1]:8891,inet:[::1]:8893
      -o cleanup_service_name=cleanup2
      -o smtpd_delay_reject=no
      dnsblog unix - - - - 0 dnsblog
      tlsproxy unix - - - - 0 tlsproxy
      pickup fifo n - - 60 1 pickup
      cleanup unix n - - - 0 cleanup
      cleanup2 unix n - - - 0 cleanup
      -o header_checks=pcre:${map}/header_checks.pcre,regexp:${map}/add_header.regexp
      -o body_checks=pcre:${map}/body_checks.pcre
      qmgr fifo n - n 300 1 qmgr
      tlsmgr unix - - - 1000? 1 tlsmgr
      rewrite unix - - - - - trivial-rewrite
      bounce unix - - - - 0 bounce
      defer unix - - - - 0 bounce
      trace unix - - - - 0 bounce
      verify unix - - - - 1 verify
      flush unix n - - 1000? 0 flush
      proxymap unix - - n - - proxymap
      proxywrite
      unix - - n - - proxymap
      smtp unix - - - - - smtp
      relay unix - - - - - smtp
      showq unix n - - - - showq
      error unix - - - - - error
      discard unix - - - - - discard
      local unix - n n - - local
      virtual unix - n n - - virtual
      lmtp unix - - - - - lmtp
      anvil unix - - - - 1 anvil
      scache unix - - - - 1 scache
      retry unix - - - - - error
      disclaimer
      unix - n n - - pipe
      flags=Rq user=filter argv=${config_directory}/filter/add_disclaimer.sh -f ${sender} -- ${recipient}
      127.0.0.1:smtp
      inet n - - - - smtpd
      -o mynetworks=127.0.0.0/8
      -o smtpd_recipient_restrictions=${localhost_smtpd_recipient_restrictions}
      193.239.107.53:submission
      inet n - - - - smtpd
      -o syslog_name=postfix:587
      -o milter_macro_daemon_name=ORIGINATING
      -o smtpd_banner=${smtpd_submission_banner}
      -o myhostname=mailout.deltaweb.de
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_tls_cert_file=/etc/ssl/certs/mailout_deltaweb_de.crt
      -o smtpd_tls_key_file=/etc/ssl/private/mailout_deltaweb_de.key
      -o smtpd_tls_CAfile=/etc/ssl/certs/ca-certificates.crt
      -o smtpd_tls_security_level=encrypt
      -o always_add_missing_headers=yes
      -o smtpd_relay_restrictions=${submission_smtpd_relay_restrictions}
      -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
      -o smtpd_reject_footer=
      -o smtpd_tls_dh1024_param_file=${config_directory}/dh_1024.pem
      193.239.107.53:smtps
      inet n - - - - smtpd
      -o syslog_name=postfix:465
      -o milter_macro_daemon_name=ORIGINATING
      -o myhostname=mailout.deltaweb.de
      -o smtpd_sasl_auth_enable=yes
      -o smtpd_tls_cert_file=/etc/ssl/certs/mailout_deltaweb_de.crt
      -o smtpd_tls_key_file=/etc/ssl/private/mailout_deltaweb_de.key
      -o smtpd_tls_CAfile=/etc/ssl/certs/ca-certificates.crt
      -o smtpd_tls_wrappermode=yes
      -o smtpd_tls_security_level=encrypt
      -o always_add_missing_headers=yes
      -o smtpd_relay_restrictions=${submission_smtpd_relay_restrictions}
      -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
      -o smtpd_reject_footer=
      -o smtpd_tls_dh1024_param_file=${config_directory}/dh_1024.pem

      -- end of postfinger output --
      --
      [*] sys4 AG

      http://sys4.de, +49 (89) 30 90 46 64
      Franziskanerstraße 15, 81669 München

      Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
      Vorstand: Patrick Ben Koetter, Marc Schiffbauer
      Aufsichtsratsvorsitzender: Florian Kirstein
    • Wietse Venema
      ... While Milters see SMTP commands as they happen, the first Milter inspects the message content and makes changes after the entire message has been received,
      Message 2 of 6 , Apr 27, 2014
        Christian R??ner:
        > I thought mail would arrive on smtpd where all the milters are
        > called and afterwards the mail would be handed over to cleanup,
        > which does canonical stuff. But it seems I am wrong :)

        While Milters see SMTP commands as they happen, the first Milter
        inspects the message content and makes changes after the entire
        message has been received, then the second Milter inspects the
        content and makes its changes, and so on.

        After the entire message is received by Postfix, the message exists
        only in the Postfix queue file. There is no place where messages
        hang around between smtpd(8) and cleanup(8), and I wasn't
        going to change that for Milters.

        Wietse
      • Christian Rößner
        Hi, ... Does it mean I can not do canonicalization with Postfix when using milters? The only solution I see is to forward mail to a second Postfix instance,
        Message 3 of 6 , Apr 27, 2014
          Hi,

          >> I thought mail would arrive on smtpd where all the milters are
          >> called and afterwards the mail would be handed over to cleanup,
          >> which does canonical stuff. But it seems I am wrong :)
          >
          > While Milters see SMTP commands as they happen, the first Milter
          > inspects the message content and makes changes after the entire
          > message has been received, then the second Milter inspects the
          > content and makes its changes, and so on.
          >
          > After the entire message is received by Postfix, the message exists
          > only in the Postfix queue file. There is no place where messages
          > hang around between smtpd(8) and cleanup(8), and I wasn't
          > going to change that for Milters.


          Does it mean I can not do canonicalization with Postfix when using milters?

          The only solution I see is to forward mail to a second Postfix instance, after mail has passed milters. And that second instance would do canonical. But this is really complicated just to have postsrsd working. I fear: too much work

          Thanks

          -Christian Rößner

          --
          [*] sys4 AG

          http://sys4.de, +49 (89) 30 90 46 64
          Franziskanerstraße 15, 81669 München

          Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
          Vorstand: Patrick Ben Koetter, Marc Schiffbauer
          Aufsichtsratsvorsitzender: Florian Kirstein
        • Alexandre Ellert
          ... Hi, Maybe you should give a try to srs-milter : https://github.com/driskell/srs-milter Alexandre Le 28 avr. 2014 à 08:33, Christian Rößner
          Message 4 of 6 , Apr 28, 2014

            Le 28 avr. 2014 à 08:33, Christian Rößner <cr@...> a écrit :

            Does it mean I can not do canonicalization with Postfix when using milters?

            The only solution I see is to forward mail to a second Postfix instance, after mail has passed milters. And that second instance would do canonical. But this is really complicated just to have postsrsd working. I fear: too much work

            Hi,

            Maybe you should give a try to srs-milter : https://github.com/driskell/srs-milter

            Alexandre
          • Christian Rößner
            ... Thank you. I had a closer look at it. Currently I try to understand, how this milter works. Not sure, if it solves my task. If I can not get it to work, I
            Message 5 of 6 , Apr 28, 2014
              >> Does it mean I can not do canonicalization with Postfix when using milters?
              >>
              >> The only solution I see is to forward mail to a second Postfix instance, after mail has passed milters. And that second instance would do canonical. But this is really complicated just to have postsrsd working. I fear: too much work
              >
              > Maybe you should give a try to srs-milter : https://github.com/driskell/srs-milter

              Thank you. I had a closer look at it. Currently I try to understand, how this milter works. Not sure, if it solves my task.

              If I can not get it to work, I contact out customers to stop using forwarding. They shall use POP3 or IMAP4 accounts.

              Thanks

              -Christian Rößner

              --
              [*] sys4 AG

              http://sys4.de, +49 (89) 30 90 46 64
              Franziskanerstraße 15, 81669 München

              Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
              Vorstand: Patrick Ben Koetter, Marc Schiffbauer
              Aufsichtsratsvorsitzender: Florian Kirstein
            • Wietse Venema
              ... srs-milter is supposed to be the tool that changes the envelope to avoid SPF errors. Postfix also supports socketmap because some SRS solutions use that.
              Message 6 of 6 , Apr 28, 2014
                Christian R??ner:
                > >> Does it mean I can not do canonicalization with Postfix when using milters?
                > >>
                > >> The only solution I see is to forward mail to a second Postfix instance, after mail has passed milters. And that second instance would do canonical. But this is really complicated just to have postsrsd working. I fear: too much work
                > >
                > > Maybe you should give a try to srs-milter : https://github.com/driskell/srs-milter
                >
                > Thank you. I had a closer look at it. Currently I try to understand, how this milter works. Not sure, if it solves my task.

                srs-milter is supposed to be the tool that changes the envelope to
                avoid SPF errors. Postfix also supports socketmap because some SRS
                solutions use that.

                Wietse
              Your message has been successfully submitted and would be delivered to recipients shortly.