You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vishal Verma <vi...@exadatum.com> on 2019/09/26 06:53:52 UTC
Urgent : Changes required in the archive
Hi All,
The help request sent by me on 20th Dec 2017 to apache spark developers
need to be edited as it contains our clients sensitive information about
the infrastructure and user Id. I understand the policies of ASF and regret
the mistake made earlier. The mistake has been highlighted to the client as
well and this might lead to repercussions for the company and me in
particular seeing the support to help correct my mistake.
Link to the prior content of the mail/archive :
http://mail-archives.apache.org/mod_mbox/spark-dev/201712.mbox/%3CCAJAL6iof9=CHLba+7pvoVfTCRKP0_7D3bcb8w05VG6TqSK_0Cg@mail.gmail.com%3E
You can replace the following content of the error log with the earlier as
I have* replaced *all required information from the content, keeping the
error as it is.
> 17/12/20 11:07:16 INFO executor.CoarseGrainedExecutorBackend: Started
> daemon with process name: 19581@*localhost*
> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for TERM
> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for HUP
> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for INT
> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls to: yarn,
> *user-me*
> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls to:
> yarn,*user-me*
> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls groups to:
> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls groups
> to:
> 17/12/20 11:07:16 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(yarn, user-me); groups with view permissions: Set(); users with modify
> permissions: Set(yarn, user-me); groups with modify permissions: Set()
> 17/12/20 11:07:16 INFO client.TransportClientFactory: Successfully created
> connection to *localhost*:35617 after 48 ms (0 ms spent in bootstraps)
> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls to:
> yarn,user-me
> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls to:
> yarn,user-me
> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls groups to:
> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls groups
> to:
> 17/12/20 11:07:17 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(yarn, user-me); groups with view permissions: Set(); users with modify
> permissions: Set(yarn, user-me); groups with modify permissions: Set()
> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully created
> connection to 127.0.0.1:35617 after 1 ms (0 ms spent in bootstraps)
> 17/12/20 11:07:17 INFO storage.DiskBlockManager: Created local directory
> at
> /hadoop/yarn/nm-local-dir/usercache/user-me/appcache/application_1512677738429_16167/blockmgr-d585ecec-829a-432b-a8f1-89503359510e
> 17/12/20 11:07:17 INFO memory.MemoryStore: MemoryStore started with
> capacity 7.8 GB
> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend: Connecting
> to driver: spark://CoarseGrainedScheduler@*localhost*:35617
> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend: Successfully
> registered with driver
> 17/12/20 11:07:17 INFO executor.Executor: Starting executor ID 1 on host
> localhost
> 17/12/20 11:07:17 INFO util.Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60054.
> 17/12/20 11:07:17 INFO netty.NettyBlockTransferService: Server created on
> *localhost*:60054
> 17/12/20 11:07:17 INFO storage.BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registering
> BlockManager BlockManagerId(1, *localhost*, 60054, None)
> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registered BlockManager
> BlockManagerId(1, *localhost*, 60054, None)
> 17/12/20 11:07:17 INFO storage.BlockManager: external shuffle service port
> = 7337
> 17/12/20 11:07:17 INFO storage.BlockManager: Registering executor with
> local external shuffle service.
> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully created
> connection to *localhost/127.0.0.1 <http://127.0.0.1>:*7337 after 1 ms (0
> ms spent in bootstraps)
> 17/12/20 11:07:17 INFO storage.BlockManager: Initialized BlockManager:
> BlockManagerId(1, *localhost*, 60054, None)
> 17/12/20 11:08:21 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED
> SIGNAL TERM
> 17/12/20 11:08:21 INFO storage.DiskBlockManager: Shutdown hook called
> 17/12/20 11:08:21 INFO util.ShutdownHookManager: Shutdown hook called
I have gone through the policies for the archives and understand the
efforts to change the archives while keeping the integrity.
I will be very thankful to you, if you understand the problem I am going
though and help me in correcting my mistake.
--
Thanks & Regards,
*Vishal Verma*
*BigData Engineer* | Exadatum Software Services Pvt. Ltd.
[image: www.exadatum.com] <http://www.exadatum.com>
Mob: +91- 967-331-3735
Pune | India
--
*
*
*DISCLAIMER:*
All the content in email is intended for the recipient
and not to be published elsewhere without Exadatum consent. And attachments
shall be send only if required and with ownership of the sender. This
message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your system. Email transmission cannot be guaranteed to be
secure or error-free, as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The sender,
therefore, does not accept liability for any errors or omissions in the
contents of this message which arise as a result of email transmission. If
verification is required, please request a hard-copy version.
Re: Urgent : Changes required in the archive
Posted by Sean Owen <sr...@gmail.com>.
The message in question has already been public, and copied to mirrors the
ASF does not control, for a year and a half.
There is a process for requesting modification to ASF archives, but this
case does not qualify:
https://www.apache.org/foundation/public-archives.html
On Thu, Sep 26, 2019 at 1:54 AM Vishal Verma <vi...@exadatum.com>
wrote:
> Hi All,
>
> The help request sent by me on 20th Dec 2017 to apache spark developers
> need to be edited as it contains our clients sensitive information about
> the infrastructure and user Id. I understand the policies of ASF and regret
> the mistake made earlier. The mistake has been highlighted to the client as
> well and this might lead to repercussions for the company and me in
> particular seeing the support to help correct my mistake.
> Link to the prior content of the mail/archive :
> http://mail-archives.apache.org/mod_mbox/spark-dev/201712.mbox/%3CCAJAL6iof9=CHLba+7pvoVfTCRKP0_7D3bcb8w05VG6TqSK_0Cg@mail.gmail.com%3E
>
>
> You can replace the following content of the error log with the earlier as
> I have* replaced *all required information from the content, keeping the
> error as it is.
>
>
>> 17/12/20 11:07:16 INFO executor.CoarseGrainedExecutorBackend: Started
>> daemon with process name: 19581@*localhost*
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for
>> TERM
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for HUP
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for INT
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls to: yarn,
>> *user-me*
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls to:
>> yarn,*user-me*
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls groups
>> to:
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls groups
>> to:
>> 17/12/20 11:07:16 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(yarn, user-me); groups with view permissions: Set(); users with modify
>> permissions: Set(yarn, user-me); groups with modify permissions: Set()
>> 17/12/20 11:07:16 INFO client.TransportClientFactory: Successfully
>> created connection to *localhost*:35617 after 48 ms (0 ms spent in
>> bootstraps)
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls to:
>> yarn,user-me
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls to:
>> yarn,user-me
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls groups
>> to:
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls groups
>> to:
>> 17/12/20 11:07:17 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(yarn, user-me); groups with view permissions: Set(); users with modify
>> permissions: Set(yarn, user-me); groups with modify permissions: Set()
>> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully
>> created connection to 127.0.0.1:35617 after 1 ms (0 ms spent in
>> bootstraps)
>> 17/12/20 11:07:17 INFO storage.DiskBlockManager: Created local directory
>> at
>> /hadoop/yarn/nm-local-dir/usercache/user-me/appcache/application_1512677738429_16167/blockmgr-d585ecec-829a-432b-a8f1-89503359510e
>> 17/12/20 11:07:17 INFO memory.MemoryStore: MemoryStore started with
>> capacity 7.8 GB
>> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend: Connecting
>> to driver: spark://CoarseGrainedScheduler@*localhost*:35617
>> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend:
>> Successfully registered with driver
>> 17/12/20 11:07:17 INFO executor.Executor: Starting executor ID 1 on host
>> localhost
>> 17/12/20 11:07:17 INFO util.Utils: Successfully started service
>> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60054.
>> 17/12/20 11:07:17 INFO netty.NettyBlockTransferService: Server created on
>> *localhost*:60054
>> 17/12/20 11:07:17 INFO storage.BlockManager: Using
>> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
>> policy
>> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registering
>> BlockManager BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registered
>> BlockManager BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:07:17 INFO storage.BlockManager: external shuffle service
>> port = 7337
>> 17/12/20 11:07:17 INFO storage.BlockManager: Registering executor with
>> local external shuffle service.
>> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully
>> created connection to *localhost/127.0.0.1 <http://127.0.0.1>:*7337
>> after 1 ms (0 ms spent in bootstraps)
>> 17/12/20 11:07:17 INFO storage.BlockManager: Initialized BlockManager:
>> BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:08:21 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED
>> SIGNAL TERM
>> 17/12/20 11:08:21 INFO storage.DiskBlockManager: Shutdown hook called
>> 17/12/20 11:08:21 INFO util.ShutdownHookManager: Shutdown hook called
>
>
>
> I have gone through the policies for the archives and understand the
> efforts to change the archives while keeping the integrity.
> I will be very thankful to you, if you understand the problem I am going
> though and help me in correcting my mistake.
>
> --
> Thanks & Regards,
>
> *Vishal Verma*
> *BigData Engineer* | Exadatum Software Services Pvt. Ltd.
> [image: www.exadatum.com] <http://www.exadatum.com>
> Mob: +91- 967-331-3735
> Pune | India
>
>
>
> *DISCLAIMER:*
> All the content in email is intended for the recipient and not to be
> published elsewhere without Exadatum consent. And attachments shall be send
> only if required and with ownership of the sender. This message contains
> confidential information and is intended only for the individual named. If
> you are not the named addressee, you should not disseminate, distribute or
> copy this email. Please notify the sender immediately by email if you have
> received this email by mistake and delete this email from your system.
> Email transmission cannot be guaranteed to be secure or error-free, as
> information could be intercepted, corrupted, lost, destroyed, arrive late
> or incomplete, or contain viruses. The sender, therefore, does not accept
> liability for any errors or omissions in the contents of this message which
> arise as a result of email transmission. If verification is required,
> please request a hard-copy version.
>
Re: Urgent : Changes required in the archive
Posted by Sean Owen <sr...@gmail.com>.
The message in question has already been public, and copied to mirrors the
ASF does not control, for a year and a half.
There is a process for requesting modification to ASF archives, but this
case does not qualify:
https://www.apache.org/foundation/public-archives.html
On Thu, Sep 26, 2019 at 1:54 AM Vishal Verma <vi...@exadatum.com>
wrote:
> Hi All,
>
> The help request sent by me on 20th Dec 2017 to apache spark developers
> need to be edited as it contains our clients sensitive information about
> the infrastructure and user Id. I understand the policies of ASF and regret
> the mistake made earlier. The mistake has been highlighted to the client as
> well and this might lead to repercussions for the company and me in
> particular seeing the support to help correct my mistake.
> Link to the prior content of the mail/archive :
> http://mail-archives.apache.org/mod_mbox/spark-dev/201712.mbox/%3CCAJAL6iof9=CHLba+7pvoVfTCRKP0_7D3bcb8w05VG6TqSK_0Cg@mail.gmail.com%3E
>
>
> You can replace the following content of the error log with the earlier as
> I have* replaced *all required information from the content, keeping the
> error as it is.
>
>
>> 17/12/20 11:07:16 INFO executor.CoarseGrainedExecutorBackend: Started
>> daemon with process name: 19581@*localhost*
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for
>> TERM
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for HUP
>> 17/12/20 11:07:16 INFO util.SignalUtils: Registered signal handler for INT
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls to: yarn,
>> *user-me*
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls to:
>> yarn,*user-me*
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing view acls groups
>> to:
>> 17/12/20 11:07:16 INFO spark.SecurityManager: Changing modify acls groups
>> to:
>> 17/12/20 11:07:16 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(yarn, user-me); groups with view permissions: Set(); users with modify
>> permissions: Set(yarn, user-me); groups with modify permissions: Set()
>> 17/12/20 11:07:16 INFO client.TransportClientFactory: Successfully
>> created connection to *localhost*:35617 after 48 ms (0 ms spent in
>> bootstraps)
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls to:
>> yarn,user-me
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls to:
>> yarn,user-me
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing view acls groups
>> to:
>> 17/12/20 11:07:17 INFO spark.SecurityManager: Changing modify acls groups
>> to:
>> 17/12/20 11:07:17 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(yarn, user-me); groups with view permissions: Set(); users with modify
>> permissions: Set(yarn, user-me); groups with modify permissions: Set()
>> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully
>> created connection to 127.0.0.1:35617 after 1 ms (0 ms spent in
>> bootstraps)
>> 17/12/20 11:07:17 INFO storage.DiskBlockManager: Created local directory
>> at
>> /hadoop/yarn/nm-local-dir/usercache/user-me/appcache/application_1512677738429_16167/blockmgr-d585ecec-829a-432b-a8f1-89503359510e
>> 17/12/20 11:07:17 INFO memory.MemoryStore: MemoryStore started with
>> capacity 7.8 GB
>> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend: Connecting
>> to driver: spark://CoarseGrainedScheduler@*localhost*:35617
>> 17/12/20 11:07:17 INFO executor.CoarseGrainedExecutorBackend:
>> Successfully registered with driver
>> 17/12/20 11:07:17 INFO executor.Executor: Starting executor ID 1 on host
>> localhost
>> 17/12/20 11:07:17 INFO util.Utils: Successfully started service
>> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60054.
>> 17/12/20 11:07:17 INFO netty.NettyBlockTransferService: Server created on
>> *localhost*:60054
>> 17/12/20 11:07:17 INFO storage.BlockManager: Using
>> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
>> policy
>> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registering
>> BlockManager BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:07:17 INFO storage.BlockManagerMaster: Registered
>> BlockManager BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:07:17 INFO storage.BlockManager: external shuffle service
>> port = 7337
>> 17/12/20 11:07:17 INFO storage.BlockManager: Registering executor with
>> local external shuffle service.
>> 17/12/20 11:07:17 INFO client.TransportClientFactory: Successfully
>> created connection to *localhost/127.0.0.1 <http://127.0.0.1>:*7337
>> after 1 ms (0 ms spent in bootstraps)
>> 17/12/20 11:07:17 INFO storage.BlockManager: Initialized BlockManager:
>> BlockManagerId(1, *localhost*, 60054, None)
>> 17/12/20 11:08:21 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED
>> SIGNAL TERM
>> 17/12/20 11:08:21 INFO storage.DiskBlockManager: Shutdown hook called
>> 17/12/20 11:08:21 INFO util.ShutdownHookManager: Shutdown hook called
>
>
>
> I have gone through the policies for the archives and understand the
> efforts to change the archives while keeping the integrity.
> I will be very thankful to you, if you understand the problem I am going
> though and help me in correcting my mistake.
>
> --
> Thanks & Regards,
>
> *Vishal Verma*
> *BigData Engineer* | Exadatum Software Services Pvt. Ltd.
> [image: www.exadatum.com] <http://www.exadatum.com>
> Mob: +91- 967-331-3735
> Pune | India
>
>
>
> *DISCLAIMER:*
> All the content in email is intended for the recipient and not to be
> published elsewhere without Exadatum consent. And attachments shall be send
> only if required and with ownership of the sender. This message contains
> confidential information and is intended only for the individual named. If
> you are not the named addressee, you should not disseminate, distribute or
> copy this email. Please notify the sender immediately by email if you have
> received this email by mistake and delete this email from your system.
> Email transmission cannot be guaranteed to be secure or error-free, as
> information could be intercepted, corrupted, lost, destroyed, arrive late
> or incomplete, or contain viruses. The sender, therefore, does not accept
> liability for any errors or omissions in the contents of this message which
> arise as a result of email transmission. If verification is required,
> please request a hard-copy version.
>